2024-09-11T04:44:51.266 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T04:44:51.281 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T04:44:51.366 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176 branch: wip-hemanth-testing-2024-09-10-0723 description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest}} email: hyelloji@redhat.com first_in_suite: false job_id: '7900176' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-hemanth-testing-2024-09-10-0723 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-hemanth-testing-2024-09-10-0723 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 owner: scheduled_hyelloji@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 9167 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-hemanth-testing-2024-09-10-0723 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOe7ROfMJxWpBwQdrxM7C8tBXKZSqsD8EtBS5Absx6gjHqK7dedvV+XC56VLQX7PXFeMagZPNpaVqwUxeSGcDA= tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:16:14 tube: smithi user: hyelloji verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T04:44:51.366 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa; will attempt to use it 2024-09-11T04:44:51.367 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks 2024-09-11T04:44:51.367 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T04:44:51.368 INFO:teuthology.task.internal:Checking packages... 2024-09-11T04:44:51.390 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8f219c476cb4f26fc5719c661c0323719a45cb30' 2024-09-11T04:44:51.390 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T04:44:51.390 INFO:teuthology.packaging:ref: None 2024-09-11T04:44:51.390 INFO:teuthology.packaging:tag: None 2024-09-11T04:44:51.390 INFO:teuthology.packaging:branch: wip-hemanth-testing-2024-09-10-0723 2024-09-11T04:44:51.390 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:44:51.390 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-hemanth-testing-2024-09-10-0723 2024-09-11T04:44:51.539 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4854.g8f219c47 2024-09-11T04:44:51.541 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T04:44:51.547 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T04:44:51.547 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T04:44:51.565 INFO:teuthology.task.internal:Saving configuration 2024-09-11T04:44:51.579 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T04:44:51.586 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T04:44:51.612 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 04:40:16.160707', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOe7ROfMJxWpBwQdrxM7C8tBXKZSqsD8EtBS5Absx6gjHqK7dedvV+XC56VLQX7PXFeMagZPNpaVqwUxeSGcDA='} 2024-09-11T04:44:51.613 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T04:44:51.617 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-11T04:44:51.618 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T04:44:51.659 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa3461e3be0>, signals=[15]) 2024-09-11T04:44:51.659 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T04:44:51.666 INFO:teuthology.task.internal:Opening connections... 2024-09-11T04:44:51.666 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T04:44:51.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:44:51.746 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T04:44:51.757 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-09-11T04:44:51.827 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-09-11T04:44:51.827 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-09-11T04:44:51.885 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-09-11T04:44:51.885 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-09-11T04:44:51.885 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-09-11T04:44:51.885 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-09-11T04:44:51.885 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-09-11T04:44:51.885 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-09-11T04:44:51.886 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T04:44:51.886 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-09-11T04:44:51.886 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-09-11T04:44:51.886 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T04:44:51.886 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-09-11T04:44:51.886 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T04:44:51.886 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T04:44:51.886 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T04:44:51.887 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-09-11T04:44:51.910 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T04:44:51.921 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T04:44:51.989 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T04:44:51.989 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-09-11T04:44:52.006 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T04:44:52.014 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T04:44:52.014 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-09-11T04:44:52.075 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T04:44:52.110 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T04:44:52.125 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T04:44:52.125 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T04:44:52.127 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T04:44:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:44:52.145 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-11T04:44:52.204 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-503.el9.x86_64 2024-09-11T04:44:52.204 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-503.el9.x86_64 2024-09-11T04:44:52.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-09-11T04:44:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-11T04:45:04.706 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-11T04:45:10.298 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-11T04:45:11.707 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 115 kB/s | 19 kB 00:00 2024-09-11T04:45:12.561 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-11T04:45:18.208 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-11T04:45:20.290 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T04:45:20.290 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T04:45:20.377 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:45:20.384 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:45:20.385 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-09-11T04:45:20.385 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:45:20.385 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:45:20.385 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T04:45:20.386 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:45:20.386 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T04:45:20.386 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T04:45:20.386 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T04:45:20.387 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:45:20.387 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:45:20.387 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:45:20.387 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2024-09-11T04:45:20.387 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:45:20.388 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 79 M 2024-09-11T04:45:20.388 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 126 M 2024-09-11T04:45:20.388 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:45:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 242 kB/s | 23 kB 00:00 2024-09-11T04:45:21.109 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-11T04:45:21.484 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 38 MB/s | 29 MB 00:00 2024-09-11T04:45:21.785 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-09-11T04:45:21.785 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:45:21.785 INFO:teuthology.orchestra.run.smithi031.stdout:Total 56 MB/s | 79 MB 00:01 2024-09-11T04:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:45:22.881 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:45:22.881 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:45:23.905 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:45:23.905 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:45:26.476 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:45:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T04:45:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T04:45:30.043 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T04:45:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T04:45:35.291 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T04:45:35.332 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T04:45:40.469 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T04:46:19.364 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T04:46:19.405 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T04:46:19.786 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T04:46:19.787 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T04:46:19.787 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T04:46:19.787 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:46:20.586 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:46:20.943 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T04:46:21.035 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T04:46:21.035 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T04:46:21.036 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T04:46:21.036 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T04:46:21.036 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T04:46:21.036 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T04:46:21.036 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T04:46:21.036 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T04:46:21.095 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T04:46:21.096 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T04:46:21.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2024-09-11T04:46:21.140 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2024-09-11T04:46:21.140 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2024-09-11T04:46:21.140 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2024-09-11T04:46:21.140 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 82.el9 2024-09-11T04:46:21.140 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8274670 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2024-09-11T04:46:21.141 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2024-09-11T04:46:21.142 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:46:21.142 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T04:46:21.142 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T04:46:21.142 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T04:46:21.142 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2024-09-11T04:46:21.142 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:46:21.142 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T04:46:21.143 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T04:46:21.143 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-505.el9.x86_64 2024-09-11T04:46:21.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T04:46:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-09-11T04:46:23.146 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T04:46:23.174 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-09-11T04:46:23.176 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-11T04:46:23.191 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.7JPUwOBKBR 2024-09-11T04:46:23.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7JPUwOBKBR 2024-09-11T04:46:23.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.7JPUwOBKBR 2024-09-11T04:46:23.395 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.7JPUwOBKBR is 6KB 2024-09-11T04:46:23.444 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.7JPUwOBKBR 2024-09-11T04:46:23.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T04:46:23.521 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T04:46:23.521 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T04:46:23.521 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T04:46:23.521 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T04:46:23.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T04:46:23.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-09-11T04:46:53.693 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T04:46:53.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T04:46:53.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:47:53.757 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T04:48:02.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T04:48:02.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:48:03.145 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-11T04:48:03.507 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T04:48:03.507 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T04:48:03.508 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-11T04:48:03.525 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-505.el9.x86_64 2024-09-11T04:48:03.525 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T04:48:03.525 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T04:48:03.525 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T04:48:04.526 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T04:48:04.527 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T04:48:04.594 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-09-11T04:48:04.618 DEBUG:teuthology.parallel:result is None 2024-09-11T04:48:04.619 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T04:48:04.632 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T04:48:04.632 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T04:48:04.650 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T04:48:04.656 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T04:48:04.668 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T04:48:04.669 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T04:48:04.734 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T04:48:04.744 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T04:48:04.744 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T04:48:04.804 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T04:48:04.816 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T04:48:04.818 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T04:48:04.829 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T04:48:04.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T04:48:04.888 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T04:48:04.900 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T04:48:04.900 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T04:48:04.946 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T04:48:05.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T04:48:05.081 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T04:48:05.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T04:48:05.208 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:48:05.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T04:48:05.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-09-11T04:48:05.363 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T04:48:05.521 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T04:48:05.529 INFO:teuthology.task.internal:Starting timer... 2024-09-11T04:48:05.529 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T04:48:05.566 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T04:48:05.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-09-11T04:48:05.619 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-09-11T04:48:05.621 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T04:48:05.621 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-09-11T04:48:05.689 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-09-11T04:48:05.689 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive'} 2024-09-11T04:48:05.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T04:48:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:48:05.757 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T04:48:05.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-09-11T04:48:05.836 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T04:48:05.851 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T04:48:05.863 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T04:48:05.864 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T04:51:51.939 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com')] 2024-09-11T04:51:51.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T04:51:51.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:51:52.024 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-11T04:51:52.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T04:51:52.118 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T04:51:52.129 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T04:51:52.129 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T04:51:52.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T04:51:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T04:51:52.204 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T04:51:52.231 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-09-11T04:51:52.243 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-09-11T04:51:52.258 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T04:51:52.273 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T04:51:52.334 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-09-11T04:51:52.338 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T04:51:52.338 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-09-11T04:51:52.339 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T04:51:52.339 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T04:51:52.339 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T04:51:52.339 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T04:51:52.340 INFO:teuthology.run_tasks:Running task install... 2024-09-11T04:51:52.348 DEBUG:teuthology.task.install:project ceph 2024-09-11T04:51:52.348 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T04:51:52.348 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-11T04:51:52.348 INFO:teuthology.task.install:Using flavor: default 2024-09-11T04:51:52.353 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T04:51:52.353 INFO:teuthology.task.install:extra packages: [] 2024-09-11T04:51:52.354 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-11T04:51:52.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:51:52.498 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/ 2024-09-11T04:51:52.498 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4854.g8f219c47 2024-09-11T04:51:52.612 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T04:51:52.612 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:51:52.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T04:51:52.640 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T04:51:52.641 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8f219c476cb4f26fc5719c661c0323719a45cb30/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T04:51:52.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T04:51:52.787 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T04:51:52.852 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-09-11T04:51:52.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-09-11T04:51:53.092 INFO:teuthology.orchestra.run.smithi031.stdout:45 files removed 2024-09-11T04:51:53.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2024-09-11T04:51:53.710 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 290 kB/s | 84 kB 00:00 2024-09-11T04:51:54.103 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 69 kB/s | 24 kB 00:00 2024-09-11T04:51:54.386 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-11T04:51:54.853 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-11T04:51:58.238 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 20 MB 00:02 2024-09-11T04:52:02.605 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-11T04:52:04.081 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-11T04:52:05.343 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-11T04:52:11.007 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-11T04:52:13.479 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:52:13.490 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 11 M 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.5 M 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T04:52:13.491 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 22 M 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 25 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 755 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 507 k 2024-09-11T04:52:13.492 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 45 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 143 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 161 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 321 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 306 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 99 k 2024-09-11T04:52:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout:Install 31 Packages 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2024-09-11T04:52:13.494 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:13.495 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 62 M 2024-09-11T04:52:13.495 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:52:15.025 INFO:teuthology.orchestra.run.smithi031.stdout:(1/33): ceph-base-19.3.0-4854.g8f219c47.el9.x86 4.0 MB/s | 5.5 MB 00:01 2024-09-11T04:52:15.075 INFO:teuthology.orchestra.run.smithi031.stdout:(2/33): ceph-selinux-19.3.0-4854.g8f219c47.el9. 504 kB/s | 25 kB 00:00 2024-09-11T04:52:15.167 INFO:teuthology.orchestra.run.smithi031.stdout:(3/33): libcephfs2-19.3.0-4854.g8f219c47.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-11T04:52:15.242 INFO:teuthology.orchestra.run.smithi031.stdout:(4/33): libradosstriper1-19.3.0-4854.g8f219c47. 6.6 MB/s | 507 kB 00:00 2024-09-11T04:52:15.459 INFO:teuthology.orchestra.run.smithi031.stdout:(5/33): ceph-radosgw-19.3.0-4854.g8f219c47.el9. 6.1 MB/s | 11 MB 00:01 2024-09-11T04:52:15.501 INFO:teuthology.orchestra.run.smithi031.stdout:(6/33): python3-ceph-argparse-19.3.0-4854.g8f21 1.1 MB/s | 45 kB 00:00 2024-09-11T04:52:15.560 INFO:teuthology.orchestra.run.smithi031.stdout:(7/33): python3-ceph-common-19.3.0-4854.g8f219c 2.4 MB/s | 143 kB 00:00 2024-09-11T04:52:15.626 INFO:teuthology.orchestra.run.smithi031.stdout:(8/33): python3-cephfs-19.3.0-4854.g8f219c47.el 2.4 MB/s | 161 kB 00:00 2024-09-11T04:52:15.702 INFO:teuthology.orchestra.run.smithi031.stdout:(9/33): librgw2-19.3.0-4854.g8f219c47.el9.x86_6 12 MB/s | 5.5 MB 00:00 2024-09-11T04:52:15.727 INFO:teuthology.orchestra.run.smithi031.stdout:(10/33): python3-rados-19.3.0-4854.g8f219c47.el 3.1 MB/s | 321 kB 00:00 2024-09-11T04:52:15.752 INFO:teuthology.orchestra.run.smithi031.stdout:(11/33): python3-rbd-19.3.0-4854.g8f219c47.el9. 6.0 MB/s | 306 kB 00:00 2024-09-11T04:52:15.777 INFO:teuthology.orchestra.run.smithi031.stdout:(12/33): python3-rgw-19.3.0-4854.g8f219c47.el9. 1.9 MB/s | 99 kB 00:00 2024-09-11T04:52:16.020 INFO:teuthology.orchestra.run.smithi031.stdout:(13/33): ceph-common-19.3.0-4854.g8f219c47.el9. 9.2 MB/s | 22 MB 00:02 2024-09-11T04:52:16.387 INFO:teuthology.orchestra.run.smithi031.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 169 kB/s | 107 kB 00:00 2024-09-11T04:52:16.478 INFO:teuthology.orchestra.run.smithi031.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 243 kB/s | 171 kB 00:00 2024-09-11T04:52:16.504 INFO:teuthology.orchestra.run.smithi031.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-11T04:52:16.562 INFO:teuthology.orchestra.run.smithi031.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 295 kB/s | 160 kB 00:00 2024-09-11T04:52:16.671 INFO:teuthology.orchestra.run.smithi031.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-09-11T04:52:16.788 INFO:teuthology.orchestra.run.smithi031.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-09-11T04:52:16.880 INFO:teuthology.orchestra.run.smithi031.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 1.6 MB/s | 662 kB 00:00 2024-09-11T04:52:16.905 INFO:teuthology.orchestra.run.smithi031.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 307 kB/s | 36 kB 00:00 2024-09-11T04:52:16.963 INFO:teuthology.orchestra.run.smithi031.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 755 kB/s | 221 kB 00:00 2024-09-11T04:52:16.997 INFO:teuthology.orchestra.run.smithi031.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-09-11T04:52:17.022 INFO:teuthology.orchestra.run.smithi031.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-09-11T04:52:17.064 INFO:teuthology.orchestra.run.smithi031.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-11T04:52:17.090 INFO:teuthology.orchestra.run.smithi031.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 288 kB/s | 26 kB 00:00 2024-09-11T04:52:17.106 INFO:teuthology.orchestra.run.smithi031.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-11T04:52:17.140 INFO:teuthology.orchestra.run.smithi031.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-11T04:52:17.182 INFO:teuthology.orchestra.run.smithi031.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-11T04:52:17.265 INFO:teuthology.orchestra.run.smithi031.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-11T04:52:17.333 INFO:teuthology.orchestra.run.smithi031.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-11T04:52:17.508 INFO:teuthology.orchestra.run.smithi031.stdout:(32/33): librados2-19.3.0-4854.g8f219c47.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-11T04:52:17.558 INFO:teuthology.orchestra.run.smithi031.stdout:(33/33): librbd1-19.3.0-4854.g8f219c47.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-11T04:52:17.563 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:52:17.563 INFO:teuthology.orchestra.run.smithi031.stdout:Total 15 MB/s | 62 MB 00:04 2024-09-11T04:52:17.682 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:52:17.723 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:52:17.723 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:52:18.437 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:52:18.437 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:52:20.040 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:52:20.137 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-11T04:52:20.171 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-11T04:52:20.213 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-11T04:52:20.236 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-11T04:52:20.260 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-11T04:52:20.457 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-11T04:52:20.519 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-11T04:52:20.869 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-11T04:52:21.002 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T04:52:21.104 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T04:52:21.197 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 8/35 2024-09-11T04:52:21.224 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/35 2024-09-11T04:52:21.295 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/35 2024-09-11T04:52:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-11T04:52:21.377 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9. 11/35 2024-09-11T04:52:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 12/35 2024-09-11T04:52:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 13/35 2024-09-11T04:52:21.541 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 13/35 2024-09-11T04:52:21.575 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-11T04:52:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-11T04:52:21.718 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-11T04:52:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-11T04:52:21.790 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-11T04:52:21.820 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-11T04:52:21.853 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T04:52:21.897 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-11T04:52:22.239 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-11T04:52:22.291 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 23/35 2024-09-11T04:52:22.383 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 23/35 2024-09-11T04:52:22.413 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 24/35 2024-09-11T04:52:22.449 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-11T04:52:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-11T04:52:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 27/35 2024-09-11T04:52:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 27/35 2024-09-11T04:52:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 28/35 2024-09-11T04:52:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x8 29/35 2024-09-11T04:52:25.585 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T04:52:25.616 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T04:52:26.231 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T04:52:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 31/35 2024-09-11T04:52:26.363 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 31/35 2024-09-11T04:52:26.363 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T04:52:26.363 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T04:52:26.363 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:26.396 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T04:52:36.621 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T04:52:36.621 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-09-11T04:52:36.622 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-09-11T04:52:36.622 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-09-11T04:52:36.622 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-09-11T04:52:36.622 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-09-11T04:52:36.622 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-09-11T04:52:36.622 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-09-11T04:52:36.622 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:37.650 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 33/35 2024-09-11T04:52:37.677 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 33/35 2024-09-11T04:52:37.678 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T04:52:37.678 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T04:52:37.678 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T04:52:37.678 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T04:52:37.678 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:37.679 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T04:52:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T04:52:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T04:52:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T04:52:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 1/35 2024-09-11T04:52:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2/35 2024-09-11T04:52:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 3/35 2024-09-11T04:52:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 4/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 5/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 6/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9. 8/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x8 9/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 10/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 11/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 12/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 13/35 2024-09-11T04:52:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-11T04:52:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-11T04:52:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-11T04:52:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-11T04:52:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-11T04:52:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-11T04:52:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-11T04:52:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-11T04:52:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-11T04:52:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T04:52:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-11T04:52:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 34/35 2024-09-11T04:52:40.096 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T04:52:40.096 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T04:52:40.097 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.098 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:40.099 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:52:40.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2024-09-11T04:52:41.008 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:31 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:52:41.147 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:52:41.147 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:41.147 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 50 M 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 162 k 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:41.148 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2024-09-11T04:52:41.149 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:41.149 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 51 M 2024-09-11T04:52:41.149 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 217 M 2024-09-11T04:52:41.149 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:52:41.417 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): libcephsqlite-19.3.0-4854.g8f219c47.el9. 737 kB/s | 162 kB 00:00 2024-09-11T04:52:41.476 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-09-11T04:52:41.584 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 381 kB/s | 64 kB 00:00 2024-09-11T04:52:42.751 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): ceph-test-19.3.0-4854.g8f219c47.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-11T04:52:42.753 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:52:42.754 INFO:teuthology.orchestra.run.smithi031.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-11T04:52:42.782 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:52:42.805 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:52:42.805 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:52:43.214 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:52:43.215 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:52:43.770 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:52:43.870 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T04:52:43.909 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T04:52:43.936 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 3/4 2024-09-11T04:52:48.980 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 3/4 2024-09-11T04:52:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 4/4 2024-09-11T04:52:51.389 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 4/4 2024-09-11T04:52:51.389 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 1/4 2024-09-11T04:52:51.389 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2/4 2024-09-11T04:52:51.389 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T04:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T04:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:52:51.753 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T04:52:51.753 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T04:52:51.753 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:51.753 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:52:52.024 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2024-09-11T04:52:52.616 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:42 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:52:52.763 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 6.6 k 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 2.4 M 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 1.5 M 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 291 k 2024-09-11T04:52:52.766 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 4.8 M 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 17 M 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T04:52:52.767 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T04:52:52.768 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout:Install 37 Packages 2024-09-11T04:52:52.769 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:52:52.770 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 M 2024-09-11T04:52:52.770 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 109 M 2024-09-11T04:52:52.770 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:52:53.793 INFO:teuthology.orchestra.run.smithi031.stdout:(1/37): ceph-19.3.0-4854.g8f219c47.el9.x86_64.r 50 kB/s | 6.6 kB 00:00 2024-09-11T04:52:54.042 INFO:teuthology.orchestra.run.smithi031.stdout:(2/37): ceph-mds-19.3.0-4854.g8f219c47.el9.x86_ 6.2 MB/s | 2.4 MB 00:00 2024-09-11T04:52:54.085 INFO:teuthology.orchestra.run.smithi031.stdout:(3/37): ceph-mgr-19.3.0-4854.g8f219c47.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2024-09-11T04:52:54.151 INFO:teuthology.orchestra.run.smithi031.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4854.g8f21 4.3 MB/s | 291 kB 00:00 2024-09-11T04:52:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-11T04:52:54.435 INFO:teuthology.orchestra.run.smithi031.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-09-11T04:52:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-11T04:52:54.552 INFO:teuthology.orchestra.run.smithi031.stdout:(8/37): ceph-mon-19.3.0-4854.g8f219c47.el9.x86_ 6.3 MB/s | 4.8 MB 00:00 2024-09-11T04:52:54.711 INFO:teuthology.orchestra.run.smithi031.stdout:(9/37): ceph-osd-19.3.0-4854.g8f219c47.el9.x86_ 26 MB/s | 17 MB 00:00 2024-09-11T04:52:54.737 INFO:teuthology.orchestra.run.smithi031.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 540 kB/s | 135 kB 00:00 2024-09-11T04:52:54.786 INFO:teuthology.orchestra.run.smithi031.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-11T04:52:54.862 INFO:teuthology.orchestra.run.smithi031.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 404 kB/s | 125 kB 00:00 2024-09-11T04:52:54.946 INFO:teuthology.orchestra.run.smithi031.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 903 kB/s | 188 kB 00:00 2024-09-11T04:52:54.971 INFO:teuthology.orchestra.run.smithi031.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-09-11T04:52:54.997 INFO:teuthology.orchestra.run.smithi031.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 822 kB/s | 172 kB 00:00 2024-09-11T04:52:55.022 INFO:teuthology.orchestra.run.smithi031.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-11T04:52:55.047 INFO:teuthology.orchestra.run.smithi031.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-09-11T04:52:55.072 INFO:teuthology.orchestra.run.smithi031.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 507 kB/s | 51 kB 00:00 2024-09-11T04:52:55.114 INFO:teuthology.orchestra.run.smithi031.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-11T04:52:55.148 INFO:teuthology.orchestra.run.smithi031.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-09-11T04:52:55.173 INFO:teuthology.orchestra.run.smithi031.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-11T04:52:55.198 INFO:teuthology.orchestra.run.smithi031.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-11T04:52:55.223 INFO:teuthology.orchestra.run.smithi031.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 709 kB/s | 18 kB 00:00 2024-09-11T04:52:55.278 INFO:teuthology.orchestra.run.smithi031.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 430 kB/s | 23 kB 00:00 2024-09-11T04:52:55.307 INFO:teuthology.orchestra.run.smithi031.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-09-11T04:52:55.332 INFO:teuthology.orchestra.run.smithi031.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 414 kB/s | 19 kB 00:00 2024-09-11T04:52:55.357 INFO:teuthology.orchestra.run.smithi031.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-11T04:52:55.383 INFO:teuthology.orchestra.run.smithi031.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-11T04:52:55.408 INFO:teuthology.orchestra.run.smithi031.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.6 MB/s | 79 kB 00:00 2024-09-11T04:52:55.433 INFO:teuthology.orchestra.run.smithi031.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 654 kB/s | 16 kB 00:00 2024-09-11T04:52:55.458 INFO:teuthology.orchestra.run.smithi031.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-11T04:52:55.484 INFO:teuthology.orchestra.run.smithi031.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-11T04:52:55.509 INFO:teuthology.orchestra.run.smithi031.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-11T04:52:55.534 INFO:teuthology.orchestra.run.smithi031.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-11T04:52:55.559 INFO:teuthology.orchestra.run.smithi031.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-11T04:52:55.585 INFO:teuthology.orchestra.run.smithi031.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-11T04:52:55.602 INFO:teuthology.orchestra.run.smithi031.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-09-11T04:52:55.606 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:52:55.606 INFO:teuthology.orchestra.run.smithi031.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-11T04:52:55.759 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:52:55.803 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:52:55.804 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:52:56.282 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:52:56.282 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:52:57.680 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:52:57.746 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T04:52:57.780 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T04:52:57.815 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T04:52:57.892 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T04:52:57.940 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T04:52:57.973 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T04:52:58.070 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T04:52:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T04:52:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T04:52:58.236 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T04:52:58.265 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T04:52:58.291 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T04:52:58.324 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T04:52:58.374 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T04:52:58.412 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T04:52:58.446 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T04:52:58.504 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T04:52:58.543 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T04:52:58.620 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T04:52:58.770 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T04:52:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T04:52:58.905 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T04:52:58.943 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T04:52:59.001 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T04:52:59.073 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T04:52:59.268 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T04:52:59.313 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T04:52:59.394 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T04:52:59.483 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T04:52:59.532 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T04:52:59.672 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T04:52:59.851 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9. 32/37 2024-09-11T04:52:59.880 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 33/37 2024-09-11T04:52:59.907 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 33/37 2024-09-11T04:52:59.908 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T04:52:59.908 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T04:52:59.908 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T04:52:59.908 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T04:52:59.908 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:01.456 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 34/37 2024-09-11T04:53:01.485 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 34/37 2024-09-11T04:53:01.485 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T04:53:01.485 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T04:53:01.485 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T04:53:01.485 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T04:53:01.486 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:01.956 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 35/37 2024-09-11T04:53:01.981 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 35/37 2024-09-11T04:53:01.982 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T04:53:01.982 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T04:53:01.982 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T04:53:01.982 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T04:53:01.982 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:02.223 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 36/37 2024-09-11T04:53:02.249 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 36/37 2024-09-11T04:53:02.249 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T04:53:02.249 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T04:53:02.249 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T04:53:02.250 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T04:53:02.250 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:02.317 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 37/37 2024-09-11T04:53:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 37/37 2024-09-11T04:53:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 1/37 2024-09-11T04:53:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2/37 2024-09-11T04:53:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 3/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 4/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 5/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9. 6/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T04:53:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T04:53:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T04:53:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T04:53:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T04:53:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T04:53:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T04:53:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T04:53:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T04:53:04.931 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T04:53:04.931 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:04.931 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:04.931 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:53:04.931 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:53:04.931 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T04:53:04.932 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T04:53:04.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:04.934 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:05.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2024-09-11T04:53:05.860 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:05.954 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:05.997 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:05.999 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:05.999 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:06.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2024-09-11T04:53:06.640 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 783 k 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:06.782 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 783 k 2024-09-11T04:53:06.783 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 790 k 2024-09-11T04:53:06.783 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:07.031 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-19.3.0-4854.g8f219c47.el9.noarch.rpm 3.1 MB/s | 783 kB 00:00 2024-09-11T04:53:07.031 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:07.031 INFO:teuthology.orchestra.run.smithi031.stdout:Total 3.1 MB/s | 783 kB 00:00 2024-09-11T04:53:07.032 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:07.037 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:07.037 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:07.043 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:07.043 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:07.121 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:07.495 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T04:53:07.579 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T04:53:07.952 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T04:53:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T04:53:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:08.187 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:08.187 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:08.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T04:53:08.990 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:58 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:09.128 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:09.129 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:09.129 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-09-11T04:53:09.129 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:09.129 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 145 k 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 145 k 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 432 k 2024-09-11T04:53:09.130 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:09.319 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-19.3.0-4854.g8f219c 768 kB/s | 145 kB 00:00 2024-09-11T04:53:09.319 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:09.320 INFO:teuthology.orchestra.run.smithi031.stdout:Total 762 kB/s | 145 kB 00:00 2024-09-11T04:53:09.320 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:09.328 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:09.328 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:09.389 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:09.389 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:09.534 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:09.595 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T04:53:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T04:53:09.621 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T04:53:09.621 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T04:53:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T04:53:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:10.570 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:53:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:10.571 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:10.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2024-09-11T04:53:11.393 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:11.487 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:11.531 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:11.532 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:11.532 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:11.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T04:53:12.186 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:02 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:12.327 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:12.328 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:12.328 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:53:12.328 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 14 M 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 34 k 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 16 k 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T04:53:12.329 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T04:53:12.330 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T04:53:12.330 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T04:53:12.330 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:12.330 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:12.330 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:12.330 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2024-09-11T04:53:12.330 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:12.331 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 19 M 2024-09-11T04:53:12.331 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 159 M 2024-09-11T04:53:12.331 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:13.005 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4854.g8f 232 kB/s | 34 kB 00:00 2024-09-11T04:53:13.031 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4854.g8f2 95 kB/s | 16 kB 00:00 2024-09-11T04:53:13.514 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4854.g8f219c4 22 MB/s | 14 MB 00:00 2024-09-11T04:53:13.873 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-11T04:53:13.915 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 305 kB/s | 269 kB 00:00 2024-09-11T04:53:13.957 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 6.4 MB/s | 548 kB 00:00 2024-09-11T04:53:13.982 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-09-11T04:53:14.016 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.3 MB/s | 144 kB 00:00 2024-09-11T04:53:14.041 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-11T04:53:14.074 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): protobuf-3.14.0-13.el9.x86_64.rpm 967 kB/s | 1.0 MB 00:01 2024-09-11T04:53:14.100 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-11T04:53:14.159 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-09-11T04:53:14.163 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:14.163 INFO:teuthology.orchestra.run.smithi031.stdout:Total 11 MB/s | 19 MB 00:01 2024-09-11T04:53:14.217 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:14.239 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:14.239 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:14.527 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:14.528 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:15.212 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-11T04:53:15.253 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-11T04:53:15.301 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-11T04:53:15.338 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-11T04:53:15.447 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-11T04:53:15.585 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-11T04:53:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-11T04:53:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-11T04:53:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-11T04:53:15.795 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9 10/12 2024-09-11T04:53:19.166 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el 11/12 2024-09-11T04:53:19.224 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 12/12 2024-09-11T04:53:21.162 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 12/12 2024-09-11T04:53:21.162 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el 1/12 2024-09-11T04:53:21.162 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 2/12 2024-09-11T04:53:21.162 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9 3/12 2024-09-11T04:53:21.163 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-11T04:53:21.163 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-11T04:53:21.163 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-11T04:53:21.163 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-11T04:53:21.163 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-11T04:53:21.163 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-11T04:53:21.163 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-11T04:53:21.163 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-11T04:53:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-11T04:53:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:21.484 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T04:53:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:21.485 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:21.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T04:53:22.331 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:12 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:22.470 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:22.471 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-11T04:53:22.471 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:53:22.471 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-11T04:53:22.471 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:22.471 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 7.4 M 2024-09-11T04:53:22.471 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:22.472 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:22.473 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-11T04:53:22.473 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2024-09-11T04:53:22.473 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:22.473 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 38 M 2024-09-11T04:53:22.473 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 212 M 2024-09-11T04:53:22.473 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:22.680 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-11T04:53:22.722 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-11T04:53:22.814 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 33 kB 00:00 2024-09-11T04:53:22.864 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-11T04:53:22.915 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 767 kB/s | 38 kB 00:00 2024-09-11T04:53:23.107 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.8 MB/s | 3.0 MB 00:00 2024-09-11T04:53:23.165 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-11T04:53:23.241 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-09-11T04:53:23.307 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 250 kB 00:00 2024-09-11T04:53:23.400 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 5.1 MB 00:00 2024-09-11T04:53:23.792 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-11T04:53:24.651 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): ceph-mgr-diskprediction-local-19.3.0-4 3.5 MB/s | 7.4 MB 00:02 2024-09-11T04:53:24.651 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:24.652 INFO:teuthology.orchestra.run.smithi031.stdout:Total 18 MB/s | 38 MB 00:02 2024-09-11T04:53:25.017 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:25.044 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:25.045 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:25.516 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:25.517 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:26.304 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:26.386 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T04:53:26.424 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T04:53:26.529 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T04:53:26.567 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T04:53:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T04:53:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T04:53:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T04:53:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T04:53:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T04:53:29.005 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T04:53:30.081 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T04:53:30.106 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 12/12 2024-09-11T04:53:32.616 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 12/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 1/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T04:53:32.617 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T04:53:33.034 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T04:53:33.034 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:33.034 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:33.034 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:33.034 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T04:53:33.034 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:33.035 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:33.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2024-09-11T04:53:34.024 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:34.164 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:34.165 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:53:34.165 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:34.165 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 49 k 2024-09-11T04:53:34.165 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:53:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T04:53:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T04:53:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:34.167 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:34.167 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:34.167 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2024-09-11T04:53:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:34.167 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.2 M 2024-09-11T04:53:34.167 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2024-09-11T04:53:34.167 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:34.522 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): ceph-mgr-rook-19.3.0-4854.g8f219c47.el9 285 kB/s | 49 kB 00:00 2024-09-11T04:53:34.639 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 66 kB/s | 19 kB 00:00 2024-09-11T04:53:34.664 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 81 kB/s | 26 kB 00:00 2024-09-11T04:53:35.073 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 366 kB/s | 159 kB 00:00 2024-09-11T04:53:35.173 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 341 kB/s | 222 kB 00:00 2024-09-11T04:53:35.223 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 357 kB/s | 54 kB 00:00 2024-09-11T04:53:35.249 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 477 kB/s | 279 kB 00:00 2024-09-11T04:53:35.282 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 296 kB/s | 32 kB 00:00 2024-09-11T04:53:35.316 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 157 kB/s | 14 kB 00:00 2024-09-11T04:53:35.349 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-11T04:53:35.375 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-11T04:53:35.399 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-09-11T04:53:35.441 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-11T04:53:35.445 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:35.445 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-11T04:53:35.518 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:35.535 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:35.535 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:35.633 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:35.634 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T04:53:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T04:53:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T04:53:36.485 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T04:53:36.527 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T04:53:36.634 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T04:53:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T04:53:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T04:53:37.555 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T04:53:37.600 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T04:53:37.634 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T04:53:37.683 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T04:53:37.712 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 13/13 2024-09-11T04:53:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 13/13 2024-09-11T04:53:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 1/13 2024-09-11T04:53:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T04:53:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T04:53:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T04:53:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T04:53:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T04:53:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T04:53:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T04:53:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T04:53:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T04:53:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T04:53:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T04:53:38.765 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T04:53:38.765 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:38.765 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:38.765 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:38.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T04:53:38.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T04:53:38.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T04:53:38.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:38.766 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:39.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T04:53:39.604 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:29 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:39.745 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:39.745 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:39.745 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:53:39.745 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:39.746 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:39.746 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 154 k 2024-09-11T04:53:39.746 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:53:39.746 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T04:53:39.746 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T04:53:39.746 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T04:53:39.747 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T04:53:39.747 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T04:53:39.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:39.747 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:39.747 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:39.747 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2024-09-11T04:53:39.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:39.748 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 7.0 M 2024-09-11T04:53:39.748 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 33 M 2024-09-11T04:53:39.748 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:40.242 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-11T04:53:40.292 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-11T04:53:40.317 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4854.g8f219c47.e 690 kB/s | 154 kB 00:00 2024-09-11T04:53:40.343 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-11T04:53:40.376 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-11T04:53:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-09-11T04:53:41.613 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:41.613 INFO:teuthology.orchestra.run.smithi031.stdout:Total 3.8 MB/s | 7.0 MB 00:01 2024-09-11T04:53:41.690 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:41.703 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:41.703 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:41.808 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:41.809 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:42.167 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:42.357 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T04:53:42.404 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T04:53:42.958 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T04:53:43.051 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T04:53:43.127 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T04:53:43.156 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 6/6 2024-09-11T04:53:43.929 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 6/6 2024-09-11T04:53:43.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/6 2024-09-11T04:53:43.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T04:53:43.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T04:53:43.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T04:53:43.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T04:53:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T04:53:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:44.216 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:44.465 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2024-09-11T04:53:45.050 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:45.188 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:45.189 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:53:45.189 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:45.189 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 900 k 2024-09-11T04:53:45.189 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T04:53:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T04:53:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:45.190 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:45.190 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:45.190 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2024-09-11T04:53:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:45.190 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 980 k 2024-09-11T04:53:45.190 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 2.9 M 2024-09-11T04:53:45.190 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:45.436 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 556 kB/s | 80 kB 00:00 2024-09-11T04:53:45.603 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-19.3.0-4854.g8f219c47.el9.x86_ 2.8 MB/s | 900 kB 00:00 2024-09-11T04:53:45.603 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:45.603 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.3 MB/s | 980 kB 00:00 2024-09-11T04:53:45.615 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:45.638 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:45.638 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:45.723 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:45.723 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:45.919 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:46.069 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T04:53:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2/2 2024-09-11T04:53:46.761 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2/2 2024-09-11T04:53:46.761 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/2 2024-09-11T04:53:47.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T04:53:47.061 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:47.061 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:47.061 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T04:53:47.061 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:47.061 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:47.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2024-09-11T04:53:47.927 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:48.068 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:48.068 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:48.068 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T04:53:48.068 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 285 k 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 285 k 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.3 M 2024-09-11T04:53:48.069 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:48.306 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-volume-19.3.0-4854.g8f219c47.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-11T04:53:48.307 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:48.307 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-11T04:53:48.307 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:48.317 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:48.317 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:48.345 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:48.345 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:48.580 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:48.650 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T04:53:48.668 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T04:53:48.668 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T04:53:48.668 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T04:53:48.668 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T04:53:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:49.662 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T04:53:49.662 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:49.662 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:49.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2024-09-11T04:53:50.546 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:50.689 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:50.689 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:50.689 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-09-11T04:53:50.689 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:50.689 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 127 k 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 127 k 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 456 k 2024-09-11T04:53:50.690 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:50.911 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-19.3.0-4854.g8f219c47.el9.x86_64 574 kB/s | 127 kB 00:00 2024-09-11T04:53:50.912 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:50.912 INFO:teuthology.orchestra.run.smithi031.stdout:Total 570 kB/s | 127 kB 00:00 2024-09-11T04:53:50.912 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:50.918 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:50.918 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:50.991 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:50.992 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:51.151 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:51.253 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:53:51.747 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:53:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:53:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:52.003 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:53:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:52.003 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:52.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2024-09-11T04:53:52.845 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:52.941 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:52.987 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:52.988 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:52.988 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:53.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2024-09-11T04:53:53.669 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:53.809 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:53.810 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:53.810 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-09-11T04:53:53.810 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:53.810 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:53:53.810 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 32 k 2024-09-11T04:53:53.810 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:53.811 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:53:53.811 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:53:53.811 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-11T04:53:53.811 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:53.811 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 32 k 2024-09-11T04:53:53.811 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 142 k 2024-09-11T04:53:53.811 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:53:53.965 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-19.3.0-4854.g8f219c47.el9.x86_6 206 kB/s | 32 kB 00:00 2024-09-11T04:53:53.965 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:53:53.966 INFO:teuthology.orchestra.run.smithi031.stdout:Total 204 kB/s | 32 kB 00:00 2024-09-11T04:53:53.966 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:53:53.970 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:53:53.970 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:53:53.988 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:53:53.988 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:53:54.082 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:53:54.199 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:53:54.606 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:53:54.844 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:53:54.844 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:54.844 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:53:54.844 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:53:54.844 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:53:54.844 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:55.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2024-09-11T04:53:55.593 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:55.686 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:55.732 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:55.733 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:55.733 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:55.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2024-09-11T04:53:56.390 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:56.484 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:56.530 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:56.532 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:56.532 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:56.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2024-09-11T04:53:57.195 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:57.290 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:57.337 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:57.338 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:57.338 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:57.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2024-09-11T04:53:57.993 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:58.088 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:58.135 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:58.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2024-09-11T04:53:58.797 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:58.893 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:58.935 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:58.936 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:58.936 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:59.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2024-09-11T04:53:59.600 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:49 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:53:59.695 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T04:53:59.741 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:53:59.742 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T04:53:59.742 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:53:59.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2024-09-11T04:54:00.405 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:54:00.542 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:54:00.543 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-09-11T04:54:00.543 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:00.543 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:54:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 85 k 2024-09-11T04:54:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:00.544 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:54:00.544 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:00.544 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-11T04:54:00.544 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:00.544 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 85 k 2024-09-11T04:54:00.544 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 231 k 2024-09-11T04:54:00.544 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:54:00.726 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-19.3.0-4854.g8f219c47.el9.x86_64.rpm 468 kB/s | 85 kB 00:00 2024-09-11T04:54:00.727 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:54:00.727 INFO:teuthology.orchestra.run.smithi031.stdout:Total 464 kB/s | 85 kB 00:00 2024-09-11T04:54:00.727 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:54:00.735 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:54:00.735 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:54:00.804 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:54:00.804 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:54:00.967 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:54:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:01.462 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:01.760 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:01.760 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:01.760 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:54:01.761 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:54:01.761 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:01.761 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:54:01.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2024-09-11T04:54:02.588 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:52 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:54:02.732 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:54:02.733 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:02.733 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-09-11T04:54:02.733 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:02.733 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:54:02.733 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T04:54:02.733 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:02.734 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:54:02.734 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:02.734 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-11T04:54:02.734 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:02.734 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.2 M 2024-09-11T04:54:02.734 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 14 M 2024-09-11T04:54:02.734 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:54:03.146 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-19.3.0-4854.g8f219c47.el9.x86_64.rpm 7.7 MB/s | 3.2 MB 00:00 2024-09-11T04:54:03.146 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:54:03.147 INFO:teuthology.orchestra.run.smithi031.stdout:Total 7.7 MB/s | 3.2 MB 00:00 2024-09-11T04:54:03.147 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:54:03.157 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:54:03.157 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:54:03.245 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:54:03.245 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:54:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:54:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:03.808 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T04:54:03.808 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T04:54:03.808 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T04:54:03.808 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T04:54:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:04.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:04.982 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:04.983 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:54:04.983 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:54:04.983 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:04.983 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:54:05.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2024-09-11T04:54:05.801 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:55 ago on Wed 11 Sep 2024 04:52:10 AM UTC. 2024-09-11T04:54:05.939 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 171 k 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:05.940 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 171 k 2024-09-11T04:54:05.941 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 490 k 2024-09-11T04:54:05.941 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T04:54:06.149 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-19.3.0-4854.g8f219c47.el9.x86_64.rpm 821 kB/s | 171 kB 00:00 2024-09-11T04:54:06.149 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T04:54:06.150 INFO:teuthology.orchestra.run.smithi031.stdout:Total 815 kB/s | 171 kB 00:00 2024-09-11T04:54:06.150 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T04:54:06.158 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T04:54:06.158 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T04:54:06.227 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T04:54:06.227 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T04:54:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T04:54:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:06.917 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:07.183 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T04:54:07.184 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:07.184 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T04:54:07.184 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T04:54:07.184 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:07.184 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T04:54:07.432 DEBUG:teuthology.parallel:result is None 2024-09-11T04:54:07.432 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:54:07.582 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T04:54:07.604 INFO:teuthology.orchestra.run.smithi031.stdout:19.3.0-4854.g8f219c47.el9 2024-09-11T04:54:07.605 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854.g8f219c47.el9 2024-09-11T04:54:07.605 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854.g8f219c47 is installed. 2024-09-11T04:54:07.607 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T04:54:07.607 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:07.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T04:54:07.671 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T04:54:07.672 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:07.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T04:54:07.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T04:54:07.803 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T04:54:07.804 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:07.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T04:54:07.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T04:54:07.934 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T04:54:07.934 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:07.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T04:54:07.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T04:54:08.065 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T04:54:08.157 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T04:54:08.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2024-09-11T04:54:08.183 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T04:54:08.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T04:54:08.249 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T04:54:08.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T04:54:08.322 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T04:54:08.323 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T04:54:08.323 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176', 'branch': 'wip-hemanth-testing-2024-09-10-0723', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest}}', 'email': 'hyelloji@redhat.com', 'first_in_suite': False, 'job_id': '7900176', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'owner': 'scheduled_hyelloji@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 9167, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-hemanth-testing-2024-09-10-0723', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOe7ROfMJxWpBwQdrxM7C8tBXKZSqsD8EtBS5Absx6gjHqK7dedvV+XC56VLQX7PXFeMagZPNpaVqwUxeSGcDA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_17:16:14', 'tube': 'smithi', 'user': 'hyelloji', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T04:54:08.323 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T04:54:08.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T04:54:08.438 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:08.438 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T04:54:08.494 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T04:54:08.494 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-09-11T04:54:08.549 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T04:54:08.549 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T04:54:08.549 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-11T04:54:08.549 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T04:54:08.550 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T04:54:08.550 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-11 04:54:04.059402232 +0000 2024-09-11T04:54:08.550 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 04:51:32.762892664 +0000 2024-09-11T04:54:08.550 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 04:51:32.762892664 +0000 2024-09-11T04:54:08.550 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-11 04:51:32.762892664 +0000 2024-09-11T04:54:08.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T04:54:08.614 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-11T04:54:08.614 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-11T04:54:08.614 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000196491 s, 2.6 MB/s 2024-09-11T04:54:08.615 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T04:54:08.673 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-09-11T04:54:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T04:54:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T04:54:08.728 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-11T04:54:08.728 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T04:54:08.729 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T04:54:08.729 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-11 04:54:04.059402232 +0000 2024-09-11T04:54:08.729 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 04:51:33.087887752 +0000 2024-09-11T04:54:08.729 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 04:51:33.087887752 +0000 2024-09-11T04:54:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-11 04:51:33.087887752 +0000 2024-09-11T04:54:08.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T04:54:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-11T04:54:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-11T04:54:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000185754 s, 2.8 MB/s 2024-09-11T04:54:08.794 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T04:54:08.850 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-09-11T04:54:08.907 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T04:54:08.907 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T04:54:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-11T04:54:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T04:54:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T04:54:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-11 04:54:04.060402216 +0000 2024-09-11T04:54:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 04:51:33.433882523 +0000 2024-09-11T04:54:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 04:51:33.433882523 +0000 2024-09-11T04:54:08.909 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-11 04:51:33.433882523 +0000 2024-09-11T04:54:08.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T04:54:08.971 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-11T04:54:08.971 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-11T04:54:08.971 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000185756 s, 2.8 MB/s 2024-09-11T04:54:08.972 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T04:54:09.029 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-09-11T04:54:09.085 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T04:54:09.085 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T04:54:09.085 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-11T04:54:09.085 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T04:54:09.085 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T04:54:09.085 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-11 04:54:04.060402216 +0000 2024-09-11T04:54:09.086 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 04:51:33.763877536 +0000 2024-09-11T04:54:09.086 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 04:51:33.763877536 +0000 2024-09-11T04:54:09.086 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-11 04:51:33.763877536 +0000 2024-09-11T04:54:09.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T04:54:09.149 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-11T04:54:09.149 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-11T04:54:09.149 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000212969 s, 2.4 MB/s 2024-09-11T04:54:09.150 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T04:54:09.207 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:54:09.207 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-11T04:54:09.207 INFO:tasks.ceph:Generating config... 2024-09-11T04:54:09.208 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T04:54:09.209 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-11T04:54:09.209 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-11T04:54:09.209 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-11T04:54:09.209 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T04:54:09.209 INFO:tasks.ceph:[global] ms type = async 2024-09-11T04:54:09.209 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T04:54:09.209 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T04:54:09.209 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T04:54:09.209 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T04:54:09.209 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T04:54:09.209 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-11T04:54:09.209 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-11T04:54:09.209 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-11T04:54:09.209 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-11T04:54:09.210 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T04:54:09.211 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-11T04:54:09.211 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-11T04:54:09.211 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T04:54:09.211 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-11T04:54:09.211 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-11T04:54:09.211 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T04:54:09.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T04:54:09.303 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T04:54:09.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T04:54:09.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T04:54:09.415 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31'), ('mon.b', '[v2:172.21.15.31:3301,v1:172.21.15.31:6790]'), ('mon.c', '[v2:172.21.15.31:3302,v1:172.21.15.31:6791]')] 2024-09-11T04:54:09.416 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],[v2:172.21.15.31:3302,v1:172.21.15.31:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-11T04:54:09.416 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:09.416 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T04:54:09.472 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --addv b '[v2:172.21.15.31:3301,v1:172.21.15.31:6790]' --addv c '[v2:172.21.15.31:3302,v1:172.21.15.31:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:54:09.560 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:54:09.560 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 843fb9a6-9086-48d4-b349-4200e697289c 2024-09-11T04:54:09.560 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 843fb9a6-9086-48d4-b349-4200e697289c 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2024-09-11T04:54:09.561427+0000 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-09-11T04:54:09.561427+0000 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 17 (quincy) 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:1: [v2:172.21.15.31:3301/0,v1:172.21.15.31:6790/0] mon.b 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:2: [v2:172.21.15.31:3302/0,v1:172.21.15.31:6791/0] mon.c 2024-09-11T04:54:09.561 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-11T04:54:09.563 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T04:54:09.578 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 843fb9a6-9086-48d4-b349-4200e697289c... 2024-09-11T04:54:09.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T04:54:09.663 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-09-11T04:54:09.664 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-09-11T04:54:09.664 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T04:54:09.664 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-09-11T04:54:09.664 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.664 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-09-11T04:54:09.664 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.665 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-09-11T04:54:09.665 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.665 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-09-11T04:54:09.665 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-09-11T04:54:09.665 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.665 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-09-11T04:54:09.665 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-09-11T04:54:09.666 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.667 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T04:54:09.668 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.668 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-09-11T04:54:09.668 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.668 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-09-11T04:54:09.668 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.668 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-09-11T04:54:09.669 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-09-11T04:54:09.669 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-09-11T04:54:09.669 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = "172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],[v2:172.21.15.31:3302,v1:172.21.15.31:6791]" 2024-09-11T04:54:09.669 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2024-09-11T04:54:09.669 INFO:teuthology.orchestra.run.smithi031.stdout: mon election default strategy = 1 2024-09-11T04:54:09.669 INFO:teuthology.orchestra.run.smithi031.stdout: ms bind msgr1 = False 2024-09-11T04:54:09.669 INFO:teuthology.orchestra.run.smithi031.stdout: ms bind msgr2 = True 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 5000 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: ms type = async 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 843fb9a6-9086-48d4-b349-4200e697289c 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-09-11T04:54:09.670 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug trim objects = true 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-09-11T04:54:09.671 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.672 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-09-11T04:54:09.672 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.672 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-09-11T04:54:09.672 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.672 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-09-11T04:54:09.672 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-09-11T04:54:09.672 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.673 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-09-11T04:54:09.673 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-09-11T04:54:09.673 INFO:teuthology.orchestra.run.smithi031.stdout: bluefs allocator = hybrid 2024-09-11T04:54:09.673 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = hybrid 2024-09-11T04:54:09.673 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2024-09-11T04:54:09.673 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2024-09-11T04:54:09.673 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore write v2 = False 2024-09-11T04:54:09.674 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore zero block detection = True 2024-09-11T04:54:09.674 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 20 2024-09-11T04:54:09.674 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 20 2024-09-11T04:54:09.674 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-11T04:54:09.674 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-09-11T04:54:09.674 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 10 2024-09-11T04:54:09.674 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T04:54:09.674 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify cached snaps = True 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify missing on start = True 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock override recovery settings = True 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: osd op queue = debug_random 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: osd op queue cut off = debug_random 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.675 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-09-11T04:54:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-11T04:54:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-09-11T04:54:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-11T04:54:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-11T04:54:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-11T04:54:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-11T04:54:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-09-11T04:54:09.676 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-11T04:54:09.677 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-09-11T04:54:09.678 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-09-11T04:54:09.678 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-11T04:54:09.678 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.678 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T04:54:09.678 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T04:54:09.678 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T04:54:09.678 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.678 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: mon scrub interval = 300 2024-09-11T04:54:09.679 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.b] 2024-09-11T04:54:09.680 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.c] 2024-09-11T04:54:09.682 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T04:54:09.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T04:54:09.779 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T04:54:09.779 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:09.779 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T04:54:09.794 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:09.794 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T04:54:09.851 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T04:54:09.851 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:09.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T04:54:09.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T04:54:09.928 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:09.928 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:54:09.983 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T04:54:09.983 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T04:54:09.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T04:54:10.076 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T04:54:10.078 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T04:54:10.079 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T04:54:10.079 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T04:54:10.079 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-11T04:54:10.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T04:54:10.104 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:54:10.104 INFO:tasks.ceph:role: osd.0 2024-09-11T04:54:10.104 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T04:54:10.104 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T04:54:10.177 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T04:54:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T04:54:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T04:54:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T04:54:10.178 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T04:54:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-11T04:54:10.178 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T04:54:10.178 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T04:54:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T04:54:10.179 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T04:54:10.680 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-11T04:54:10.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-11T04:54:10.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T04:54:10.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T04:54:10.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T04:54:10.923 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:54:10.923 INFO:tasks.ceph:role: osd.1 2024-09-11T04:54:10.923 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T04:54:10.923 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T04:54:10.998 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T04:54:10.998 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T04:54:10.998 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T04:54:10.998 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T04:54:10.999 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T04:54:10.999 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-11T04:54:10.999 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T04:54:10.999 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T04:54:10.999 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T04:54:10.999 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T04:54:11.450 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-11T04:54:11.453 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-11T04:54:11.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T04:54:11.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T04:54:11.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T04:54:11.643 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:54:11.643 INFO:tasks.ceph:role: osd.2 2024-09-11T04:54:11.643 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T04:54:11.644 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T04:54:11.717 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T04:54:11.717 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T04:54:11.718 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T04:54:11.718 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T04:54:11.718 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T04:54:11.718 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-11T04:54:11.718 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T04:54:11.718 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T04:54:11.718 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T04:54:11.718 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T04:54:12.197 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-11T04:54:12.199 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-11T04:54:12.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T04:54:12.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T04:54:12.357 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T04:54:12.423 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:54:12.423 INFO:tasks.ceph:role: osd.3 2024-09-11T04:54:12.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T04:54:12.423 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-11T04:54:12.497 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T04:54:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T04:54:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T04:54:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T04:54:12.497 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T04:54:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-11T04:54:12.498 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T04:54:12.498 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T04:54:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T04:54:12.498 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T04:54:13.020 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-11T04:54:13.023 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-11T04:54:13.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-11T04:54:13.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T04:54:13.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:54:13.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:13.296+0000 7fefeda16640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-11T04:54:13.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:13.296+0000 7fefeda16640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-11T04:54:13.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:13.296+0000 7fefeda16640 -1 bdev(0x55eb04b15800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-11T04:54:13.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:13.296+0000 7fefeda16640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-11T04:54:16.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T04:54:16.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:54:16.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:16.322+0000 7efe005c5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-11T04:54:16.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:16.322+0000 7efe005c5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-11T04:54:16.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:16.322+0000 7efe005c5640 -1 bdev(0x55aaf88d3800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-11T04:54:16.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:16.322+0000 7efe005c5640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-11T04:54:19.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T04:54:19.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:54:19.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:19.607+0000 7f5ae1a44640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-11T04:54:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:19.608+0000 7f5ae1a44640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-11T04:54:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:19.608+0000 7f5ae1a44640 -1 bdev(0x55e4dc0c3800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-11T04:54:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:19.608+0000 7f5ae1a44640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-11T04:54:23.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T04:54:23.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:54:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:23.392+0000 7fde909a2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-11T04:54:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:23.392+0000 7fde909a2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-11T04:54:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:23.392+0000 7fde909a2640 -1 bdev(0x56256da2d800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-11T04:54:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:23.393+0000 7fde909a2640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-11T04:54:26.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T04:54:26.331 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T04:54:26.331 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:26.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T04:54:26.401 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:26.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T04:54:26.470 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:26.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T04:54:26.539 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:26.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T04:54:26.606 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:26.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T04:54:26.674 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T04:54:26.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T04:54:26.740 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2024-09-11T04:54:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQByIuFmCw+aBBAADBuocmSrchBQKVvv3gv5nA== 2024-09-11T04:54:26.741 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-09-11T04:54:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB1IuFmmvivERAAoXkwpzf0SE8KuBswXfBPcw== 2024-09-11T04:54:26.741 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-09-11T04:54:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB4IuFmdnM+ExAAVMTu/3zCxS00LR9K3GfnqA== 2024-09-11T04:54:26.741 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-09-11T04:54:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB7IuFmxeBEJBAA+VaXsAIn7pFChD2D6tJOMg== 2024-09-11T04:54:26.741 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.3] 2024-09-11T04:54:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB/IuFmSAdxFxAArSKHCDQCqZ23z8G3U3w83g== 2024-09-11T04:54:26.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T04:54:26.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:54:26.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:54:26.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:54:26.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:54:27.049 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T04:54:27.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T04:54:27.076 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T04:54:27.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T04:54:27.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-11T04:54:27.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T04:54:27.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-11T04:54:27.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-11T04:54:27.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T04:54:27.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-11T04:54:27.583 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:54:27.639 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T04:54:27.639 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T04:54:27.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T04:54:27.682 INFO:tasks.ceph.mon.a:Started 2024-09-11T04:54:27.682 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T04:54:27.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T04:54:27.684 INFO:tasks.ceph.mon.b:Started 2024-09-11T04:54:27.684 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T04:54:27.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T04:54:27.686 INFO:tasks.ceph.mon.c:Started 2024-09-11T04:54:27.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph config set mgr mgr_pool false --force 2024-09-11T04:54:33.122 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T04:54:33.122 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T04:54:33.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T04:54:33.125 INFO:tasks.ceph.mgr.x:Started 2024-09-11T04:54:33.125 DEBUG:tasks.ceph:set 0 configs 2024-09-11T04:54:33.125 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T04:54:33.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T04:54:33.309 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-09-11T04:54:33.325 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T04:54:33.325 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T04:54:33.325 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:33.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T04:54:33.352 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:33.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T04:54:33.419 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:33.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T04:54:33.485 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T04:54:33.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T04:54:33.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 0a0d5d52-301d-428c-877b-eefc61f019f8 0 2024-09-11T04:54:33.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:54:33.898+0000 7fc7b91fb640 0 --2- 172.21.15.31:0/3847720028 >> [v2:172.21.15.31:3301/0,v1:172.21.15.31:6790/0] conn(0x7fc7b4091d30 0x7fc7b412b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T04:54:33.982 INFO:teuthology.orchestra.run.smithi031.stdout:0 2024-09-11T04:54:33.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 83f13be3-d367-46d4-8128-98883066e4db 1 2024-09-11T04:54:34.181 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-11T04:54:34.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 617490bf-6727-44cc-92d8-5215e439c818 2 2024-09-11T04:54:34.381 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-11T04:54:34.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 39dfc5a1-9332-4ad1-b51e-d8502c155301 3 2024-09-11T04:54:34.578 INFO:teuthology.orchestra.run.smithi031.stdout:3 2024-09-11T04:54:34.590 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T04:54:34.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T04:54:34.592 INFO:tasks.ceph.osd.0:Started 2024-09-11T04:54:34.592 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T04:54:34.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T04:54:34.594 INFO:tasks.ceph.osd.1:Started 2024-09-11T04:54:34.594 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T04:54:34.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-11T04:54:34.597 INFO:tasks.ceph.osd.2:Started 2024-09-11T04:54:34.597 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T04:54:34.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T04:54:34.599 INFO:tasks.ceph.osd.3:Started 2024-09-11T04:54:34.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:54:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:34.802 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:54:34.575381+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:54:34.813 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T04:54:34.813 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T04:54:35.215 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T04:54:35.379 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-11T04:54:35.380 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":6,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:54:34.575381+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:54:35.389 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-11T04:54:36.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T04:54:36.050+0000 7f5f4ec64640 -1 Falling back to public interface 2024-09-11T04:54:36.052 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T04:54:36.051+0000 7f80d4e24640 -1 Falling back to public interface 2024-09-11T04:54:36.060 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-11T04:54:36.059+0000 7f8e9f7dd640 -1 Falling back to public interface 2024-09-11T04:54:36.579 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T04:54:36.578+0000 7fb5f2887640 -1 Falling back to public interface 2024-09-11T04:54:38.601 INFO:tasks.ceph.mgr.x.smithi031.stderr:2024-09-11T04:54:38.601+0000 7f810eeae640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-11T04:54:38.801 INFO:tasks.ceph.mgr.x.smithi031.stderr:2024-09-11T04:54:38.801+0000 7f810eeae640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-11T04:54:39.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T04:54:39.617+0000 7f5f4ec64640 -1 osd.1 0 log_to_monitors true 2024-09-11T04:54:39.859 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T04:54:39.858+0000 7f80d4e24640 -1 osd.0 0 log_to_monitors true 2024-09-11T04:54:39.890 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-11T04:54:39.889+0000 7f8e9f7dd640 -1 osd.2 0 log_to_monitors true 2024-09-11T04:54:40.123 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T04:54:40.122+0000 7fb5f2887640 -1 osd.3 0 log_to_monitors true 2024-09-11T04:54:41.612 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-11T04:54:41.611+0000 7f8e9b76c640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T04:54:41.612 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T04:54:41.611+0000 7fb5ee816640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T04:54:41.615 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T04:54:41.614+0000 7f5f4abf3640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T04:54:41.631 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-11T04:54:41.630+0000 7f8e97593640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:54:41.632 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T04:54:41.630+0000 7fb5ea63d640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:54:41.635 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T04:54:41.634+0000 7f5f46a1a640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:54:41.793 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T04:54:42.115 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-11T04:54:42.115 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":8,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:54:41.604580+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:54:42.130 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-11T04:54:42.422 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T04:54:42.421+0000 7f80d0714640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T04:54:42.427 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T04:54:42.427+0000 7f80c8533640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:54:48.533 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T04:54:48.838 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-11T04:54:48.839 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":10,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:54:43.635281+0000","last_up_change":"2024-09-11T04:54:42.624129+0000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":199596188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":199596188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":199596188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":199596188}]},"public_addr":"172.21.15.31:6804/199596188","cluster_addr":"172.21.15.31:6805/199596188","heartbeat_back_addr":"172.21.15.31:6807/199596188","heartbeat_front_addr":"172.21.15.31:6806/199596188","state":["exists","up"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3116576774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3116576774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":3116576774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3116576774}]},"public_addr":"172.21.15.31:6800/3116576774","cluster_addr":"172.21.15.31:6801/3116576774","heartbeat_back_addr":"172.21.15.31:6803/3116576774","heartbeat_front_addr":"172.21.15.31:6802/3116576774","state":["exists","up"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2027023681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2027023681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2027023681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2027023681}]},"public_addr":"172.21.15.31:6808/2027023681","cluster_addr":"172.21.15.31:6809/2027023681","heartbeat_back_addr":"172.21.15.31:6811/2027023681","heartbeat_front_addr":"172.21.15.31:6810/2027023681","state":["exists","up"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":700800727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":700800727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":700800727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":700800727}]},"public_addr":"172.21.15.31:6812/700800727","cluster_addr":"172.21.15.31:6813/700800727","heartbeat_back_addr":"172.21.15.31:6815/700800727","heartbeat_front_addr":"172.21.15.31:6814/700800727","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.833079+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.663344+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.858324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:41.136761+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:54:48.850 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-11T04:54:48.850 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T04:54:48.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T04:54:48.851 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T04:54:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:49.163 INFO:teuthology.orchestra.run.smithi031.stdout:[{"version":3,"timestamp":"2024-09-11T04:54:42.423226+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"8635.615075"}]}] 2024-09-11T04:54:49.164 INFO:tasks.ceph_manager:config epoch is 3 2024-09-11T04:54:49.164 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T04:54:49.164 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T04:54:49.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T04:54:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:49.525 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1382329383}]},"active_addr":"172.21.15.31:6816/1382329383","active_change":"2024-09-11T04:54:37.588666+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1405964800}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1494330645}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3085784263}]}]} 2024-09-11T04:54:49.529 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T04:54:49.529 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T04:54:49.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:54:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:49.830 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:54:43.635281+0000","last_up_change":"2024-09-11T04:54:42.624129+0000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":199596188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":199596188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":199596188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":199596188}]},"public_addr":"172.21.15.31:6804/199596188","cluster_addr":"172.21.15.31:6805/199596188","heartbeat_back_addr":"172.21.15.31:6807/199596188","heartbeat_front_addr":"172.21.15.31:6806/199596188","state":["exists","up"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3116576774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3116576774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":3116576774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3116576774}]},"public_addr":"172.21.15.31:6800/3116576774","cluster_addr":"172.21.15.31:6801/3116576774","heartbeat_back_addr":"172.21.15.31:6803/3116576774","heartbeat_front_addr":"172.21.15.31:6802/3116576774","state":["exists","up"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2027023681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2027023681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2027023681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2027023681}]},"public_addr":"172.21.15.31:6808/2027023681","cluster_addr":"172.21.15.31:6809/2027023681","heartbeat_back_addr":"172.21.15.31:6811/2027023681","heartbeat_front_addr":"172.21.15.31:6810/2027023681","state":["exists","up"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":700800727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":700800727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":700800727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":700800727}]},"public_addr":"172.21.15.31:6812/700800727","cluster_addr":"172.21.15.31:6813/700800727","heartbeat_back_addr":"172.21.15.31:6815/700800727","heartbeat_front_addr":"172.21.15.31:6814/700800727","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.833079+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.663344+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.858324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:41.136761+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:54:49.846 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T04:54:49.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:54:50.147 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:50.148 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:54:43.635281+0000","last_up_change":"2024-09-11T04:54:42.624129+0000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":199596188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":199596188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":199596188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":199596188}]},"public_addr":"172.21.15.31:6804/199596188","cluster_addr":"172.21.15.31:6805/199596188","heartbeat_back_addr":"172.21.15.31:6807/199596188","heartbeat_front_addr":"172.21.15.31:6806/199596188","state":["exists","up"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3116576774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3116576774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":3116576774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3116576774}]},"public_addr":"172.21.15.31:6800/3116576774","cluster_addr":"172.21.15.31:6801/3116576774","heartbeat_back_addr":"172.21.15.31:6803/3116576774","heartbeat_front_addr":"172.21.15.31:6802/3116576774","state":["exists","up"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2027023681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2027023681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2027023681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2027023681}]},"public_addr":"172.21.15.31:6808/2027023681","cluster_addr":"172.21.15.31:6809/2027023681","heartbeat_back_addr":"172.21.15.31:6811/2027023681","heartbeat_front_addr":"172.21.15.31:6810/2027023681","state":["exists","up"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":700800727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":700800727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":700800727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":700800727}]},"public_addr":"172.21.15.31:6812/700800727","cluster_addr":"172.21.15.31:6813/700800727","heartbeat_back_addr":"172.21.15.31:6815/700800727","heartbeat_front_addr":"172.21.15.31:6814/700800727","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.833079+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.663344+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.858324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:41.136761+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:54:50.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T04:54:50.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T04:54:50.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T04:54:50.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T04:54:50.308 INFO:teuthology.orchestra.run.smithi031.stdout:38654705668 2024-09-11T04:54:50.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T04:54:50.309 INFO:teuthology.orchestra.run.smithi031.stdout:38654705668 2024-09-11T04:54:50.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:54:50.327 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-09-11T04:54:50.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:54:50.328 INFO:teuthology.orchestra.run.smithi031.stdout:38654705668 2024-09-11T04:54:50.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:54:50.633 INFO:teuthology.orchestra.run.smithi031.stdout:38654705667 2024-09-11T04:54:50.647 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.3 2024-09-11T04:54:50.648 INFO:teuthology.orchestra.run.smithi031.stdout:38654705667 2024-09-11T04:54:50.662 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.1 2024-09-11T04:54:50.707 INFO:teuthology.orchestra.run.smithi031.stdout:38654705668 2024-09-11T04:54:50.718 INFO:teuthology.orchestra.run.smithi031.stdout:38654705667 2024-09-11T04:54:50.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.0 2024-09-11T04:54:50.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.2 2024-09-11T04:54:51.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:54:51.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T04:54:51.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:54:51.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:54:51.995 INFO:teuthology.orchestra.run.smithi031.stdout:38654705668 2024-09-11T04:54:52.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.3 2024-09-11T04:54:52.009 DEBUG:teuthology.parallel:result is None 2024-09-11T04:54:52.037 INFO:teuthology.orchestra.run.smithi031.stdout:38654705668 2024-09-11T04:54:52.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.1 2024-09-11T04:54:52.051 DEBUG:teuthology.parallel:result is None 2024-09-11T04:54:52.055 INFO:teuthology.orchestra.run.smithi031.stdout:38654705669 2024-09-11T04:54:52.059 INFO:teuthology.orchestra.run.smithi031.stdout:38654705668 2024-09-11T04:54:52.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.0 2024-09-11T04:54:52.069 DEBUG:teuthology.parallel:result is None 2024-09-11T04:54:52.073 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.2 2024-09-11T04:54:52.073 DEBUG:teuthology.parallel:result is None 2024-09-11T04:54:52.073 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T04:54:52.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:54:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:52.409 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:54:52.421 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-11T04:54:51.595110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108468,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378892,"statfs":{"total":386547056640,"available":386435985408,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-11T04:54:52.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:54:52.713 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:52.714 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:54:52.726 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-11T04:54:51.595110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108468,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378892,"statfs":{"total":386547056640,"available":386435985408,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-11T04:54:52.727 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T04:54:52.727 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T04:54:52.727 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T04:54:52.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T04:54:53.063 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:53.063 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T04:54:53.075 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T04:54:53.076 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2024-09-11T04:54:53.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:54:53.404 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:53.405 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:54:43.635281+0000","last_up_change":"2024-09-11T04:54:42.624129+0000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":199596188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":199596188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":199596188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":199596188}]},"public_addr":"172.21.15.31:6804/199596188","cluster_addr":"172.21.15.31:6805/199596188","heartbeat_back_addr":"172.21.15.31:6807/199596188","heartbeat_front_addr":"172.21.15.31:6806/199596188","state":["exists","up"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3116576774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3116576774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":3116576774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3116576774}]},"public_addr":"172.21.15.31:6800/3116576774","cluster_addr":"172.21.15.31:6801/3116576774","heartbeat_back_addr":"172.21.15.31:6803/3116576774","heartbeat_front_addr":"172.21.15.31:6802/3116576774","state":["exists","up"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2027023681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2027023681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2027023681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2027023681}]},"public_addr":"172.21.15.31:6808/2027023681","cluster_addr":"172.21.15.31:6809/2027023681","heartbeat_back_addr":"172.21.15.31:6811/2027023681","heartbeat_front_addr":"172.21.15.31:6810/2027023681","state":["exists","up"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":700800727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":700800727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":700800727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":700800727}]},"public_addr":"172.21.15.31:6812/700800727","cluster_addr":"172.21.15.31:6813/700800727","heartbeat_back_addr":"172.21.15.31:6815/700800727","heartbeat_front_addr":"172.21.15.31:6814/700800727","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.833079+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.663344+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.858324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:41.136761+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:54:53.417 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2024-09-11T04:54:53.417 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-11T04:54:53.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:54:53.713 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:53.713 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:54:53.725 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-11T04:54:53.595375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108468,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378892,"statfs":{"total":386547056640,"available":386435985408,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-11T04:54:53.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:54:54.020 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:54:54.021 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:54:54.032 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-11T04:54:53.595375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108468,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378892,"statfs":{"total":386547056640,"available":386435985408,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-11T04:54:54.033 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-11T04:54:54.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2024-09-11T04:54:54.682 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2024-09-11T04:54:54.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2024-09-11T04:54:56.699 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_0' created 2024-09-11T04:54:56.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-09-11T04:54:57.696 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 1 min_size to 2 2024-09-11T04:54:57.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-11T04:54:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-11T04:54:59.737 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2024-09-11T04:54:59.737 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2024-09-11T04:54:59.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T04:54:59.800 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T04:54:59.941 INFO:teuthology.orchestra.run.smithi031.stdout:38654705671 2024-09-11T04:54:59.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T04:54:59.943 INFO:teuthology.orchestra.run.smithi031.stdout:38654705672 2024-09-11T04:54:59.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:00.300 INFO:teuthology.orchestra.run.smithi031.stdout:38654705670 2024-09-11T04:55:00.304 INFO:teuthology.orchestra.run.smithi031.stdout:38654705671 2024-09-11T04:55:00.314 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2024-09-11T04:55:00.318 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705672 got 38654705671 for osd.0 2024-09-11T04:55:01.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T04:55:01.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:01.634 INFO:teuthology.orchestra.run.smithi031.stdout:38654705670 2024-09-11T04:55:01.647 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2024-09-11T04:55:01.672 INFO:teuthology.orchestra.run.smithi031.stdout:38654705671 2024-09-11T04:55:01.685 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705672 got 38654705671 for osd.0 2024-09-11T04:55:02.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T04:55:02.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:02.969 INFO:teuthology.orchestra.run.smithi031.stdout:38654705671 2024-09-11T04:55:02.982 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705671 for osd.1 2024-09-11T04:55:02.982 DEBUG:teuthology.parallel:result is None 2024-09-11T04:55:03.005 INFO:teuthology.orchestra.run.smithi031.stdout:38654705672 2024-09-11T04:55:03.018 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705672 got 38654705672 for osd.0 2024-09-11T04:55:03.018 DEBUG:teuthology.parallel:result is None 2024-09-11T04:55:03.018 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-11T04:55:03.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:55:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:55:03.365 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-11T04:55:01.596444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108632,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378728,"statfs":{"total":386547056640,"available":386435817472,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699780+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699780+0000","last_undegraded":"2024-09-11T04:54:57.699780+0000","last_fullsized":"2024-09-11T04:54:57.699780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712973+0000","last_change":"2024-09-11T04:54:58.728939+0000","last_active":"2024-09-11T04:54:59.712973+0000","last_peered":"2024-09-11T04:54:59.712973+0000","last_clean":"2024-09-11T04:54:59.712973+0000","last_became_active":"2024-09-11T04:54:58.728520+0000","last_became_peered":"2024-09-11T04:54:58.728520+0000","last_unstale":"2024-09-11T04:54:59.712973+0000","last_undegraded":"2024-09-11T04:54:59.712973+0000","last_fullsized":"2024-09-11T04:54:59.712973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:01.026101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699012+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699012+0000","last_undegraded":"2024-09-11T04:54:57.699012+0000","last_fullsized":"2024-09-11T04:54:57.699012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712411+0000","last_change":"2024-09-11T04:54:58.718524+0000","last_active":"2024-09-11T04:54:59.712411+0000","last_peered":"2024-09-11T04:54:59.712411+0000","last_clean":"2024-09-11T04:54:59.712411+0000","last_became_active":"2024-09-11T04:54:58.718102+0000","last_became_peered":"2024-09-11T04:54:58.718102+0000","last_unstale":"2024-09-11T04:54:59.712411+0000","last_undegraded":"2024-09-11T04:54:59.712411+0000","last_fullsized":"2024-09-11T04:54:59.712411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:45.911640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699297+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699297+0000","last_undegraded":"2024-09-11T04:54:57.699297+0000","last_fullsized":"2024-09-11T04:54:57.699297+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.700260+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.700260+0000","last_undegraded":"2024-09-11T04:54:57.700260+0000","last_fullsized":"2024-09-11T04:54:57.700260+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.796789+0000","last_change":"2024-09-11T04:54:58.721845+0000","last_active":"2024-09-11T04:54:59.796789+0000","last_peered":"2024-09-11T04:54:59.796789+0000","last_clean":"2024-09-11T04:54:59.796789+0000","last_became_active":"2024-09-11T04:54:58.721192+0000","last_became_peered":"2024-09-11T04:54:58.721192+0000","last_unstale":"2024-09-11T04:54:59.796789+0000","last_undegraded":"2024-09-11T04:54:59.796789+0000","last_fullsized":"2024-09-11T04:54:59.796789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:11.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713140+0000","last_change":"2024-09-11T04:54:58.721988+0000","last_active":"2024-09-11T04:54:59.713140+0000","last_peered":"2024-09-11T04:54:59.713140+0000","last_clean":"2024-09-11T04:54:59.713140+0000","last_became_active":"2024-09-11T04:54:58.721606+0000","last_became_peered":"2024-09-11T04:54:58.721606+0000","last_unstale":"2024-09-11T04:54:59.713140+0000","last_undegraded":"2024-09-11T04:54:59.713140+0000","last_fullsized":"2024-09-11T04:54:59.713140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:11:23.562119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699506+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699506+0000","last_undegraded":"2024-09-11T04:54:57.699506+0000","last_fullsized":"2024-09-11T04:54:57.699506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699078+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699078+0000","last_undegraded":"2024-09-11T04:54:57.699078+0000","last_fullsized":"2024-09-11T04:54:57.699078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699303+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699303+0000","last_undegraded":"2024-09-11T04:54:57.699303+0000","last_fullsized":"2024-09-11T04:54:57.699303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713125+0000","last_change":"2024-09-11T04:54:58.725609+0000","last_active":"2024-09-11T04:54:59.713125+0000","last_peered":"2024-09-11T04:54:59.713125+0000","last_clean":"2024-09-11T04:54:59.713125+0000","last_became_active":"2024-09-11T04:54:58.724240+0000","last_became_peered":"2024-09-11T04:54:58.724240+0000","last_unstale":"2024-09-11T04:54:59.713125+0000","last_undegraded":"2024-09-11T04:54:59.713125+0000","last_fullsized":"2024-09-11T04:54:59.713125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:43.864014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712231+0000","last_change":"2024-09-11T04:54:58.718663+0000","last_active":"2024-09-11T04:54:59.712231+0000","last_peered":"2024-09-11T04:54:59.712231+0000","last_clean":"2024-09-11T04:54:59.712231+0000","last_became_active":"2024-09-11T04:54:58.718259+0000","last_became_peered":"2024-09-11T04:54:58.718259+0000","last_unstale":"2024-09-11T04:54:59.712231+0000","last_undegraded":"2024-09-11T04:54:59.712231+0000","last_fullsized":"2024-09-11T04:54:59.712231+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:22:02.268552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712504+0000","last_change":"2024-09-11T04:54:58.721086+0000","last_active":"2024-09-11T04:54:59.712504+0000","last_peered":"2024-09-11T04:54:59.712504+0000","last_clean":"2024-09-11T04:54:59.712504+0000","last_became_active":"2024-09-11T04:54:58.720717+0000","last_became_peered":"2024-09-11T04:54:58.720717+0000","last_unstale":"2024-09-11T04:54:59.712504+0000","last_undegraded":"2024-09-11T04:54:59.712504+0000","last_fullsized":"2024-09-11T04:54:59.712504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:27:15.351390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712646+0000","last_change":"2024-09-11T04:54:58.723900+0000","last_active":"2024-09-11T04:54:59.712646+0000","last_peered":"2024-09-11T04:54:59.712646+0000","last_clean":"2024-09-11T04:54:59.712646+0000","last_became_active":"2024-09-11T04:54:58.723511+0000","last_became_peered":"2024-09-11T04:54:58.723511+0000","last_unstale":"2024-09-11T04:54:59.712646+0000","last_undegraded":"2024-09-11T04:54:59.712646+0000","last_fullsized":"2024-09-11T04:54:59.712646+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:58:10.705170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713049+0000","last_change":"2024-09-11T04:54:58.723084+0000","last_active":"2024-09-11T04:54:59.713049+0000","last_peered":"2024-09-11T04:54:59.713049+0000","last_clean":"2024-09-11T04:54:59.713049+0000","last_became_active":"2024-09-11T04:54:58.722689+0000","last_became_peered":"2024-09-11T04:54:58.722689+0000","last_unstale":"2024-09-11T04:54:59.713049+0000","last_undegraded":"2024-09-11T04:54:59.713049+0000","last_fullsized":"2024-09-11T04:54:59.713049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:54.503975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:55:03.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:55:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:03.664 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:55:03.676 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-11T04:55:03.596683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108632,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378728,"statfs":{"total":386547056640,"available":386435817472,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699780+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699780+0000","last_undegraded":"2024-09-11T04:54:57.699780+0000","last_fullsized":"2024-09-11T04:54:57.699780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712973+0000","last_change":"2024-09-11T04:54:58.728939+0000","last_active":"2024-09-11T04:54:59.712973+0000","last_peered":"2024-09-11T04:54:59.712973+0000","last_clean":"2024-09-11T04:54:59.712973+0000","last_became_active":"2024-09-11T04:54:58.728520+0000","last_became_peered":"2024-09-11T04:54:58.728520+0000","last_unstale":"2024-09-11T04:54:59.712973+0000","last_undegraded":"2024-09-11T04:54:59.712973+0000","last_fullsized":"2024-09-11T04:54:59.712973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:01.026101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699012+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699012+0000","last_undegraded":"2024-09-11T04:54:57.699012+0000","last_fullsized":"2024-09-11T04:54:57.699012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712411+0000","last_change":"2024-09-11T04:54:58.718524+0000","last_active":"2024-09-11T04:54:59.712411+0000","last_peered":"2024-09-11T04:54:59.712411+0000","last_clean":"2024-09-11T04:54:59.712411+0000","last_became_active":"2024-09-11T04:54:58.718102+0000","last_became_peered":"2024-09-11T04:54:58.718102+0000","last_unstale":"2024-09-11T04:54:59.712411+0000","last_undegraded":"2024-09-11T04:54:59.712411+0000","last_fullsized":"2024-09-11T04:54:59.712411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:45.911640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699297+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699297+0000","last_undegraded":"2024-09-11T04:54:57.699297+0000","last_fullsized":"2024-09-11T04:54:57.699297+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.700260+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.700260+0000","last_undegraded":"2024-09-11T04:54:57.700260+0000","last_fullsized":"2024-09-11T04:54:57.700260+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.796789+0000","last_change":"2024-09-11T04:54:58.721845+0000","last_active":"2024-09-11T04:54:59.796789+0000","last_peered":"2024-09-11T04:54:59.796789+0000","last_clean":"2024-09-11T04:54:59.796789+0000","last_became_active":"2024-09-11T04:54:58.721192+0000","last_became_peered":"2024-09-11T04:54:58.721192+0000","last_unstale":"2024-09-11T04:54:59.796789+0000","last_undegraded":"2024-09-11T04:54:59.796789+0000","last_fullsized":"2024-09-11T04:54:59.796789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:11.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713140+0000","last_change":"2024-09-11T04:54:58.721988+0000","last_active":"2024-09-11T04:54:59.713140+0000","last_peered":"2024-09-11T04:54:59.713140+0000","last_clean":"2024-09-11T04:54:59.713140+0000","last_became_active":"2024-09-11T04:54:58.721606+0000","last_became_peered":"2024-09-11T04:54:58.721606+0000","last_unstale":"2024-09-11T04:54:59.713140+0000","last_undegraded":"2024-09-11T04:54:59.713140+0000","last_fullsized":"2024-09-11T04:54:59.713140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:11:23.562119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699506+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699506+0000","last_undegraded":"2024-09-11T04:54:57.699506+0000","last_fullsized":"2024-09-11T04:54:57.699506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699078+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699078+0000","last_undegraded":"2024-09-11T04:54:57.699078+0000","last_fullsized":"2024-09-11T04:54:57.699078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699303+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699303+0000","last_undegraded":"2024-09-11T04:54:57.699303+0000","last_fullsized":"2024-09-11T04:54:57.699303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713125+0000","last_change":"2024-09-11T04:54:58.725609+0000","last_active":"2024-09-11T04:54:59.713125+0000","last_peered":"2024-09-11T04:54:59.713125+0000","last_clean":"2024-09-11T04:54:59.713125+0000","last_became_active":"2024-09-11T04:54:58.724240+0000","last_became_peered":"2024-09-11T04:54:58.724240+0000","last_unstale":"2024-09-11T04:54:59.713125+0000","last_undegraded":"2024-09-11T04:54:59.713125+0000","last_fullsized":"2024-09-11T04:54:59.713125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:43.864014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712231+0000","last_change":"2024-09-11T04:54:58.718663+0000","last_active":"2024-09-11T04:54:59.712231+0000","last_peered":"2024-09-11T04:54:59.712231+0000","last_clean":"2024-09-11T04:54:59.712231+0000","last_became_active":"2024-09-11T04:54:58.718259+0000","last_became_peered":"2024-09-11T04:54:58.718259+0000","last_unstale":"2024-09-11T04:54:59.712231+0000","last_undegraded":"2024-09-11T04:54:59.712231+0000","last_fullsized":"2024-09-11T04:54:59.712231+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:22:02.268552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712504+0000","last_change":"2024-09-11T04:54:58.721086+0000","last_active":"2024-09-11T04:54:59.712504+0000","last_peered":"2024-09-11T04:54:59.712504+0000","last_clean":"2024-09-11T04:54:59.712504+0000","last_became_active":"2024-09-11T04:54:58.720717+0000","last_became_peered":"2024-09-11T04:54:58.720717+0000","last_unstale":"2024-09-11T04:54:59.712504+0000","last_undegraded":"2024-09-11T04:54:59.712504+0000","last_fullsized":"2024-09-11T04:54:59.712504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:27:15.351390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712646+0000","last_change":"2024-09-11T04:54:58.723900+0000","last_active":"2024-09-11T04:54:59.712646+0000","last_peered":"2024-09-11T04:54:59.712646+0000","last_clean":"2024-09-11T04:54:59.712646+0000","last_became_active":"2024-09-11T04:54:58.723511+0000","last_became_peered":"2024-09-11T04:54:58.723511+0000","last_unstale":"2024-09-11T04:54:59.712646+0000","last_undegraded":"2024-09-11T04:54:59.712646+0000","last_fullsized":"2024-09-11T04:54:59.712646+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:58:10.705170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713049+0000","last_change":"2024-09-11T04:54:58.723084+0000","last_active":"2024-09-11T04:54:59.713049+0000","last_peered":"2024-09-11T04:54:59.713049+0000","last_clean":"2024-09-11T04:54:59.713049+0000","last_became_active":"2024-09-11T04:54:58.722689+0000","last_became_peered":"2024-09-11T04:54:58.722689+0000","last_unstale":"2024-09-11T04:54:59.713049+0000","last_undegraded":"2024-09-11T04:54:59.713049+0000","last_fullsized":"2024-09-11T04:54:59.713049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:54.503975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:55:03.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:55:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:03.974 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:55:03.986 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-11T04:55:03.596683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108632,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378728,"statfs":{"total":386547056640,"available":386435817472,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699780+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699780+0000","last_undegraded":"2024-09-11T04:54:57.699780+0000","last_fullsized":"2024-09-11T04:54:57.699780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712973+0000","last_change":"2024-09-11T04:54:58.728939+0000","last_active":"2024-09-11T04:54:59.712973+0000","last_peered":"2024-09-11T04:54:59.712973+0000","last_clean":"2024-09-11T04:54:59.712973+0000","last_became_active":"2024-09-11T04:54:58.728520+0000","last_became_peered":"2024-09-11T04:54:58.728520+0000","last_unstale":"2024-09-11T04:54:59.712973+0000","last_undegraded":"2024-09-11T04:54:59.712973+0000","last_fullsized":"2024-09-11T04:54:59.712973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:01.026101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699012+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699012+0000","last_undegraded":"2024-09-11T04:54:57.699012+0000","last_fullsized":"2024-09-11T04:54:57.699012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712411+0000","last_change":"2024-09-11T04:54:58.718524+0000","last_active":"2024-09-11T04:54:59.712411+0000","last_peered":"2024-09-11T04:54:59.712411+0000","last_clean":"2024-09-11T04:54:59.712411+0000","last_became_active":"2024-09-11T04:54:58.718102+0000","last_became_peered":"2024-09-11T04:54:58.718102+0000","last_unstale":"2024-09-11T04:54:59.712411+0000","last_undegraded":"2024-09-11T04:54:59.712411+0000","last_fullsized":"2024-09-11T04:54:59.712411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:45.911640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699297+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699297+0000","last_undegraded":"2024-09-11T04:54:57.699297+0000","last_fullsized":"2024-09-11T04:54:57.699297+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.700260+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.700260+0000","last_undegraded":"2024-09-11T04:54:57.700260+0000","last_fullsized":"2024-09-11T04:54:57.700260+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.796789+0000","last_change":"2024-09-11T04:54:58.721845+0000","last_active":"2024-09-11T04:54:59.796789+0000","last_peered":"2024-09-11T04:54:59.796789+0000","last_clean":"2024-09-11T04:54:59.796789+0000","last_became_active":"2024-09-11T04:54:58.721192+0000","last_became_peered":"2024-09-11T04:54:58.721192+0000","last_unstale":"2024-09-11T04:54:59.796789+0000","last_undegraded":"2024-09-11T04:54:59.796789+0000","last_fullsized":"2024-09-11T04:54:59.796789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:11.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713140+0000","last_change":"2024-09-11T04:54:58.721988+0000","last_active":"2024-09-11T04:54:59.713140+0000","last_peered":"2024-09-11T04:54:59.713140+0000","last_clean":"2024-09-11T04:54:59.713140+0000","last_became_active":"2024-09-11T04:54:58.721606+0000","last_became_peered":"2024-09-11T04:54:58.721606+0000","last_unstale":"2024-09-11T04:54:59.713140+0000","last_undegraded":"2024-09-11T04:54:59.713140+0000","last_fullsized":"2024-09-11T04:54:59.713140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:11:23.562119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699506+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699506+0000","last_undegraded":"2024-09-11T04:54:57.699506+0000","last_fullsized":"2024-09-11T04:54:57.699506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699078+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699078+0000","last_undegraded":"2024-09-11T04:54:57.699078+0000","last_fullsized":"2024-09-11T04:54:57.699078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-11T04:54:57.699303+0000","last_change":"2024-09-11T04:54:56.683804+0000","last_active":"2024-09-11T04:54:56.683804+0000","last_peered":"2024-09-11T04:54:56.683804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T04:54:57.699303+0000","last_undegraded":"2024-09-11T04:54:57.699303+0000","last_fullsized":"2024-09-11T04:54:57.699303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713125+0000","last_change":"2024-09-11T04:54:58.725609+0000","last_active":"2024-09-11T04:54:59.713125+0000","last_peered":"2024-09-11T04:54:59.713125+0000","last_clean":"2024-09-11T04:54:59.713125+0000","last_became_active":"2024-09-11T04:54:58.724240+0000","last_became_peered":"2024-09-11T04:54:58.724240+0000","last_unstale":"2024-09-11T04:54:59.713125+0000","last_undegraded":"2024-09-11T04:54:59.713125+0000","last_fullsized":"2024-09-11T04:54:59.713125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:43.864014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712231+0000","last_change":"2024-09-11T04:54:58.718663+0000","last_active":"2024-09-11T04:54:59.712231+0000","last_peered":"2024-09-11T04:54:59.712231+0000","last_clean":"2024-09-11T04:54:59.712231+0000","last_became_active":"2024-09-11T04:54:58.718259+0000","last_became_peered":"2024-09-11T04:54:58.718259+0000","last_unstale":"2024-09-11T04:54:59.712231+0000","last_undegraded":"2024-09-11T04:54:59.712231+0000","last_fullsized":"2024-09-11T04:54:59.712231+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:22:02.268552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712504+0000","last_change":"2024-09-11T04:54:58.721086+0000","last_active":"2024-09-11T04:54:59.712504+0000","last_peered":"2024-09-11T04:54:59.712504+0000","last_clean":"2024-09-11T04:54:59.712504+0000","last_became_active":"2024-09-11T04:54:58.720717+0000","last_became_peered":"2024-09-11T04:54:58.720717+0000","last_unstale":"2024-09-11T04:54:59.712504+0000","last_undegraded":"2024-09-11T04:54:59.712504+0000","last_fullsized":"2024-09-11T04:54:59.712504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:27:15.351390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712646+0000","last_change":"2024-09-11T04:54:58.723900+0000","last_active":"2024-09-11T04:54:59.712646+0000","last_peered":"2024-09-11T04:54:59.712646+0000","last_clean":"2024-09-11T04:54:59.712646+0000","last_became_active":"2024-09-11T04:54:58.723511+0000","last_became_peered":"2024-09-11T04:54:58.723511+0000","last_unstale":"2024-09-11T04:54:59.712646+0000","last_undegraded":"2024-09-11T04:54:59.712646+0000","last_fullsized":"2024-09-11T04:54:59.712646+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:58:10.705170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713049+0000","last_change":"2024-09-11T04:54:58.723084+0000","last_active":"2024-09-11T04:54:59.713049+0000","last_peered":"2024-09-11T04:54:59.713049+0000","last_clean":"2024-09-11T04:54:59.713049+0000","last_became_active":"2024-09-11T04:54:58.722689+0000","last_became_peered":"2024-09-11T04:54:58.722689+0000","last_unstale":"2024-09-11T04:54:59.713049+0000","last_undegraded":"2024-09-11T04:54:59.713049+0000","last_fullsized":"2024-09-11T04:54:59.713049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:54.503975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:55:06.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:55:07.297 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:55:07.309 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-11T04:55:05.596901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108696,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378664,"statfs":{"total":386547056640,"available":386435751936,"internally_reserved":0,"allocated":573440,"data_stored":170680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713631+0000","last_change":"2024-09-11T04:54:58.726970+0000","last_active":"2024-09-11T04:54:59.713631+0000","last_peered":"2024-09-11T04:54:59.713631+0000","last_clean":"2024-09-11T04:54:59.713631+0000","last_became_active":"2024-09-11T04:54:58.725356+0000","last_became_peered":"2024-09-11T04:54:58.725356+0000","last_unstale":"2024-09-11T04:54:59.713631+0000","last_undegraded":"2024-09-11T04:54:59.713631+0000","last_fullsized":"2024-09-11T04:54:59.713631+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:40:52.194671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712973+0000","last_change":"2024-09-11T04:54:58.728939+0000","last_active":"2024-09-11T04:54:59.712973+0000","last_peered":"2024-09-11T04:54:59.712973+0000","last_clean":"2024-09-11T04:54:59.712973+0000","last_became_active":"2024-09-11T04:54:58.728520+0000","last_became_peered":"2024-09-11T04:54:58.728520+0000","last_unstale":"2024-09-11T04:54:59.712973+0000","last_undegraded":"2024-09-11T04:54:59.712973+0000","last_fullsized":"2024-09-11T04:54:59.712973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:01.026101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712827+0000","last_change":"2024-09-11T04:54:58.723234+0000","last_active":"2024-09-11T04:54:59.712827+0000","last_peered":"2024-09-11T04:54:59.712827+0000","last_clean":"2024-09-11T04:54:59.712827+0000","last_became_active":"2024-09-11T04:54:58.722861+0000","last_became_peered":"2024-09-11T04:54:58.722861+0000","last_unstale":"2024-09-11T04:54:59.712827+0000","last_undegraded":"2024-09-11T04:54:59.712827+0000","last_fullsized":"2024-09-11T04:54:59.712827+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:00:29.798555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712411+0000","last_change":"2024-09-11T04:54:58.718524+0000","last_active":"2024-09-11T04:54:59.712411+0000","last_peered":"2024-09-11T04:54:59.712411+0000","last_clean":"2024-09-11T04:54:59.712411+0000","last_became_active":"2024-09-11T04:54:58.718102+0000","last_became_peered":"2024-09-11T04:54:58.718102+0000","last_unstale":"2024-09-11T04:54:59.712411+0000","last_undegraded":"2024-09-11T04:54:59.712411+0000","last_fullsized":"2024-09-11T04:54:59.712411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:45.911640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712552+0000","last_change":"2024-09-11T04:54:58.720470+0000","last_active":"2024-09-11T04:54:59.712552+0000","last_peered":"2024-09-11T04:54:59.712552+0000","last_clean":"2024-09-11T04:54:59.712552+0000","last_became_active":"2024-09-11T04:54:58.720056+0000","last_became_peered":"2024-09-11T04:54:58.720056+0000","last_unstale":"2024-09-11T04:54:59.712552+0000","last_undegraded":"2024-09-11T04:54:59.712552+0000","last_fullsized":"2024-09-11T04:54:59.712552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:45:23.019617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713097+0000","last_change":"2024-09-11T04:54:58.723714+0000","last_active":"2024-09-11T04:54:59.713097+0000","last_peered":"2024-09-11T04:54:59.713097+0000","last_clean":"2024-09-11T04:54:59.713097+0000","last_became_active":"2024-09-11T04:54:58.723346+0000","last_became_peered":"2024-09-11T04:54:58.723346+0000","last_unstale":"2024-09-11T04:54:59.713097+0000","last_undegraded":"2024-09-11T04:54:59.713097+0000","last_fullsized":"2024-09-11T04:54:59.713097+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:55.701146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.796789+0000","last_change":"2024-09-11T04:54:58.721845+0000","last_active":"2024-09-11T04:54:59.796789+0000","last_peered":"2024-09-11T04:54:59.796789+0000","last_clean":"2024-09-11T04:54:59.796789+0000","last_became_active":"2024-09-11T04:54:58.721192+0000","last_became_peered":"2024-09-11T04:54:58.721192+0000","last_unstale":"2024-09-11T04:54:59.796789+0000","last_undegraded":"2024-09-11T04:54:59.796789+0000","last_fullsized":"2024-09-11T04:54:59.796789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:11.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713140+0000","last_change":"2024-09-11T04:54:58.721988+0000","last_active":"2024-09-11T04:54:59.713140+0000","last_peered":"2024-09-11T04:54:59.713140+0000","last_clean":"2024-09-11T04:54:59.713140+0000","last_became_active":"2024-09-11T04:54:58.721606+0000","last_became_peered":"2024-09-11T04:54:58.721606+0000","last_unstale":"2024-09-11T04:54:59.713140+0000","last_undegraded":"2024-09-11T04:54:59.713140+0000","last_fullsized":"2024-09-11T04:54:59.713140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:11:23.562119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"16'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713571+0000","last_change":"2024-09-11T04:54:58.726313+0000","last_active":"2024-09-11T04:54:59.713571+0000","last_peered":"2024-09-11T04:54:59.713571+0000","last_clean":"2024-09-11T04:54:59.713571+0000","last_became_active":"2024-09-11T04:54:58.723910+0000","last_became_peered":"2024-09-11T04:54:58.723910+0000","last_unstale":"2024-09-11T04:54:59.713571+0000","last_undegraded":"2024-09-11T04:54:59.713571+0000","last_fullsized":"2024-09-11T04:54:59.713571+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:56:31.782072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713063+0000","last_change":"2024-09-11T04:54:58.721257+0000","last_active":"2024-09-11T04:54:59.713063+0000","last_peered":"2024-09-11T04:54:59.713063+0000","last_clean":"2024-09-11T04:54:59.713063+0000","last_became_active":"2024-09-11T04:54:58.720883+0000","last_became_peered":"2024-09-11T04:54:58.720883+0000","last_unstale":"2024-09-11T04:54:59.713063+0000","last_undegraded":"2024-09-11T04:54:59.713063+0000","last_fullsized":"2024-09-11T04:54:59.713063+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:14:33.590113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713606+0000","last_change":"2024-09-11T04:54:58.724750+0000","last_active":"2024-09-11T04:54:59.713606+0000","last_peered":"2024-09-11T04:54:59.713606+0000","last_clean":"2024-09-11T04:54:59.713606+0000","last_became_active":"2024-09-11T04:54:58.724356+0000","last_became_peered":"2024-09-11T04:54:58.724356+0000","last_unstale":"2024-09-11T04:54:59.713606+0000","last_undegraded":"2024-09-11T04:54:59.713606+0000","last_fullsized":"2024-09-11T04:54:59.713606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:50:03.336539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713125+0000","last_change":"2024-09-11T04:54:58.725609+0000","last_active":"2024-09-11T04:54:59.713125+0000","last_peered":"2024-09-11T04:54:59.713125+0000","last_clean":"2024-09-11T04:54:59.713125+0000","last_became_active":"2024-09-11T04:54:58.724240+0000","last_became_peered":"2024-09-11T04:54:58.724240+0000","last_unstale":"2024-09-11T04:54:59.713125+0000","last_undegraded":"2024-09-11T04:54:59.713125+0000","last_fullsized":"2024-09-11T04:54:59.713125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:43.864014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712231+0000","last_change":"2024-09-11T04:54:58.718663+0000","last_active":"2024-09-11T04:54:59.712231+0000","last_peered":"2024-09-11T04:54:59.712231+0000","last_clean":"2024-09-11T04:54:59.712231+0000","last_became_active":"2024-09-11T04:54:58.718259+0000","last_became_peered":"2024-09-11T04:54:58.718259+0000","last_unstale":"2024-09-11T04:54:59.712231+0000","last_undegraded":"2024-09-11T04:54:59.712231+0000","last_fullsized":"2024-09-11T04:54:59.712231+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:22:02.268552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712504+0000","last_change":"2024-09-11T04:54:58.721086+0000","last_active":"2024-09-11T04:54:59.712504+0000","last_peered":"2024-09-11T04:54:59.712504+0000","last_clean":"2024-09-11T04:54:59.712504+0000","last_became_active":"2024-09-11T04:54:58.720717+0000","last_became_peered":"2024-09-11T04:54:58.720717+0000","last_unstale":"2024-09-11T04:54:59.712504+0000","last_undegraded":"2024-09-11T04:54:59.712504+0000","last_fullsized":"2024-09-11T04:54:59.712504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:27:15.351390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.712646+0000","last_change":"2024-09-11T04:54:58.723900+0000","last_active":"2024-09-11T04:54:59.712646+0000","last_peered":"2024-09-11T04:54:59.712646+0000","last_clean":"2024-09-11T04:54:59.712646+0000","last_became_active":"2024-09-11T04:54:58.723511+0000","last_became_peered":"2024-09-11T04:54:58.723511+0000","last_unstale":"2024-09-11T04:54:59.712646+0000","last_undegraded":"2024-09-11T04:54:59.712646+0000","last_fullsized":"2024-09-11T04:54:59.712646+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:58:10.705170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T04:54:59.713049+0000","last_change":"2024-09-11T04:54:58.723084+0000","last_active":"2024-09-11T04:54:59.713049+0000","last_peered":"2024-09-11T04:54:59.713049+0000","last_clean":"2024-09-11T04:54:59.713049+0000","last_became_active":"2024-09-11T04:54:58.722689+0000","last_became_peered":"2024-09-11T04:54:58.722689+0000","last_unstale":"2024-09-11T04:54:59.713049+0000","last_undegraded":"2024-09-11T04:54:59.713049+0000","last_fullsized":"2024-09-11T04:54:59.713049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:54.503975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:55:07.310 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-11T04:55:07.310 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-11T04:55:07.311 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-11T04:55:07.364 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-11T04:55:07.364 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-11T04:55:07.417 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2024-09-11T04:55:07.418 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2024-09-11T04:55:07.469 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-11T04:55:07.469 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-11T04:55:07.522 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-11T04:55:07.523 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-11T04:55:07.575 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2024-09-11T04:55:07.576 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2024-09-11T04:55:07.628 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-11T04:55:07.628 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-11T04:55:07.681 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-11T04:55:07.681 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-11T04:55:07.735 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2024-09-11T04:55:07.735 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2024-09-11T04:55:07.787 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-11T04:55:07.787 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-11T04:55:07.840 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-11T04:55:07.840 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-11T04:55:07.893 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2024-09-11T04:55:07.893 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2024-09-11T04:55:07.945 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-11T04:55:07.945 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-11T04:55:07.998 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-11T04:55:07.998 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-11T04:55:08.052 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2024-09-11T04:55:08.053 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2024-09-11T04:55:08.105 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-11T04:55:08.140 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-11T04:55:08.194 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-11T04:55:08.194 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-11T04:55:08.247 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2024-09-11T04:55:08.247 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2024-09-11T04:55:08.299 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-11T04:55:08.299 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-11T04:55:08.352 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-11T04:55:08.353 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-11T04:55:08.406 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2024-09-11T04:55:08.406 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2024-09-11T04:55:08.458 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-11T04:55:08.458 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-11T04:55:08.511 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-11T04:55:08.511 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-11T04:55:08.565 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2024-09-11T04:55:08.565 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2024-09-11T04:55:08.617 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-11T04:55:08.617 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-11T04:55:08.670 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-11T04:55:08.670 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-11T04:55:08.724 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2024-09-11T04:55:08.725 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2024-09-11T04:55:08.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-11T04:55:08.901 INFO:teuthology.orchestra.run.smithi031.stdout:{} 2024-09-11T04:55:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-11T04:55:08.912 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T04:55:08.913 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:55:08.938 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T04:55:08.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-11T04:55:09.287 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-11T04:55:09.287+0000 7f5da4bc6640 -1 mon.a@0(leader).osd e16 definitely_dead 0 2024-09-11T04:55:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:marked down osd.0. 2024-09-11T04:55:09.748 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T04:55:09.748 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:55:09.787 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T04:55:09.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-11T04:55:10.141 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-11T04:55:10.141+0000 7f5da4bc6640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-11T04:55:10.731 INFO:teuthology.orchestra.run.smithi031.stderr:marked down osd.3. 2024-09-11T04:55:10.747 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2024-09-11T04:55:10.747 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2024-09-11T04:55:11.752 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-11T04:55:11.752 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-11T04:55:11.810 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-11T04:55:11.810 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-11T04:55:11.865 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2024-09-11T04:55:11.865 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2024-09-11T04:55:11.917 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-11T04:55:11.918 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-11T04:55:11.970 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-11T04:55:11.971 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-11T04:55:12.023 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2024-09-11T04:55:12.024 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2024-09-11T04:55:12.075 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-11T04:55:12.076 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-11T04:55:12.128 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-11T04:55:12.129 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-11T04:55:12.181 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2024-09-11T04:55:12.181 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2024-09-11T04:55:12.234 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-11T04:55:12.234 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-11T04:55:12.286 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-11T04:55:12.286 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-11T04:55:12.338 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2024-09-11T04:55:12.338 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2024-09-11T04:55:12.391 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-11T04:55:12.391 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-11T04:55:12.443 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-11T04:55:12.444 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-11T04:55:12.495 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2024-09-11T04:55:12.496 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2024-09-11T04:55:12.548 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-11T04:55:12.548 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-11T04:55:12.601 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-11T04:55:12.601 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-11T04:55:12.653 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2024-09-11T04:55:12.654 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2024-09-11T04:55:12.706 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-11T04:55:12.706 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-11T04:55:12.759 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-11T04:55:12.760 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-11T04:55:12.812 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2024-09-11T04:55:12.813 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2024-09-11T04:55:12.865 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-11T04:55:12.865 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-11T04:55:12.917 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-11T04:55:12.918 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-11T04:55:12.970 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2024-09-11T04:55:12.971 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2024-09-11T04:55:13.024 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-11T04:55:13.025 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-11T04:55:13.077 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-11T04:55:13.077 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-11T04:55:13.130 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T04:55:13.130 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:55:13.160 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T04:55:13.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-11T04:55:13.508 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-11T04:55:13.508+0000 7f5da4bc6640 -1 mon.a@0(leader).osd e19 definitely_dead 0 2024-09-11T04:55:13.759 INFO:teuthology.orchestra.run.smithi031.stderr:marked down osd.1. 2024-09-11T04:55:13.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-11T04:55:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:marked osd lost in epoch 20 2024-09-11T04:55:14.772 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T04:55:14.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T04:55:14.775 INFO:tasks.ceph.osd.0:Started 2024-09-11T04:55:14.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T04:55:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-09-11T04:55:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-09-11T04:55:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-09-11T04:55:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-09-11T04:55:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-09-11T04:55:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-09-11T04:55:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:config diff 2024-09-11T04:55:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:config diff get 2024-09-11T04:55:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-09-11T04:55:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:config help [] 2024-09-11T04:55:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-09-11T04:55:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: invalid command 2024-09-11T04:55:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T04:55:14.885 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-11T04:55:15.703 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T04:55:15.703+0000 7f5a3c20f640 -1 Falling back to public interface 2024-09-11T04:55:19.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T04:55:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator fragmentation block 2024-09-11T04:55:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: invalid command 2024-09-11T04:55:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T04:55:20.000 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-11T04:55:20.006 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T04:55:20.006+0000 7f5a3c20f640 -1 osd.0 16 log_to_monitors true 2024-09-11T04:55:20.833 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T04:55:20.833+0000 7f5a2f8a5640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:55:25.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T04:55:25.122 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-11T04:55:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-11T04:55:25.122 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:25.131 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2024-09-11T04:55:25.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:55:25.431 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:25.431 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:55:22.823877+0000","last_up_change":"2024-09-11T04:55:21.822443+0000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-11T04:54:55.691815+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":448159694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":448159694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":448159694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":448159694}]},"public_addr":"172.21.15.31:6800/448159694","cluster_addr":"172.21.15.31:6801/448159694","heartbeat_back_addr":"172.21.15.31:6803/448159694","heartbeat_front_addr":"172.21.15.31:6802/448159694","state":["exists","up"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3116576774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3116576774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":3116576774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3116576774}]},"public_addr":"172.21.15.31:6800/3116576774","cluster_addr":"172.21.15.31:6801/3116576774","heartbeat_back_addr":"172.21.15.31:6803/3116576774","heartbeat_front_addr":"172.21.15.31:6802/3116576774","state":["exists"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2027023681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2027023681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2027023681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2027023681}]},"public_addr":"172.21.15.31:6808/2027023681","cluster_addr":"172.21.15.31:6809/2027023681","heartbeat_back_addr":"172.21.15.31:6811/2027023681","heartbeat_front_addr":"172.21.15.31:6810/2027023681","state":["exists","up"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":700800727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":700800727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":700800727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":700800727}]},"public_addr":"172.21.15.31:6812/700800727","cluster_addr":"172.21.15.31:6813/700800727","heartbeat_back_addr":"172.21.15.31:6815/700800727","heartbeat_front_addr":"172.21.15.31:6814/700800727","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.833079+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.663344+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.858324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:41.136761+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:55:25.444 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2024-09-11T04:55:25.444 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T04:55:25.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T04:55:25.446 INFO:tasks.ceph.osd.3:Started 2024-09-11T04:55:25.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T04:55:25.547 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:config diff 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:config diff get 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:config help [] 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-09-11T04:55:25.548 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: invalid command 2024-09-11T04:55:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T04:55:25.550 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-11T04:55:25.834 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-11T04:55:25.834+0000 7f5da73cb640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 6 pgs recovery_unfound (PG_DAMAGED) 2024-09-11T04:55:26.611 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T04:55:26.612+0000 7f5b9580b640 -1 Falling back to public interface 2024-09-11T04:55:30.375 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T04:55:30.375+0000 7f5b9580b640 -1 osd.3 17 log_to_monitors true 2024-09-11T04:55:30.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-09-11T04:55:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-09-11T04:55:30.668 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator fragmentation block 2024-09-11T04:55:30.668 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: invalid command 2024-09-11T04:55:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T04:55:30.670 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-11T04:55:30.909 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T04:55:30.909+0000 7f5b8cebe640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:55:32.775 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-11T04:55:32.776+0000 7f5da73cb640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-11T04:55:35.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T04:55:35.792 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:35.792 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-11T04:55:35.792 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-11T04:55:35.792 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:35.801 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2024-09-11T04:55:35.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:55:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:36.104 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:55:32.912119+0000","last_up_change":"2024-09-11T04:55:31.901492+0000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-11T04:54:55.691815+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":26,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":448159694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":448159694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":448159694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":448159694}]},"public_addr":"172.21.15.31:6800/448159694","cluster_addr":"172.21.15.31:6801/448159694","heartbeat_back_addr":"172.21.15.31:6803/448159694","heartbeat_front_addr":"172.21.15.31:6802/448159694","state":["exists","up"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3116576774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":3116576774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":3116576774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3116576774}]},"public_addr":"172.21.15.31:6800/3116576774","cluster_addr":"172.21.15.31:6801/3116576774","heartbeat_back_addr":"172.21.15.31:6803/3116576774","heartbeat_front_addr":"172.21.15.31:6802/3116576774","state":["exists"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2027023681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2027023681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2027023681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2027023681}]},"public_addr":"172.21.15.31:6808/2027023681","cluster_addr":"172.21.15.31:6809/2027023681","heartbeat_back_addr":"172.21.15.31:6811/2027023681","heartbeat_front_addr":"172.21.15.31:6810/2027023681","state":["exists","up"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":26,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2434669726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2434669726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":2434669726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2434669726}]},"public_addr":"172.21.15.31:6804/2434669726","cluster_addr":"172.21.15.31:6805/2434669726","heartbeat_back_addr":"172.21.15.31:6807/2434669726","heartbeat_front_addr":"172.21.15.31:6806/2434669726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.833079+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.663344+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.858324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:41.136761+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:55:36.116 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2024-09-11T04:55:36.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T04:55:36.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T04:55:36.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T04:55:36.259 INFO:teuthology.orchestra.run.smithi031.stdout:98784247813 2024-09-11T04:55:36.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:36.263 INFO:teuthology.orchestra.run.smithi031.stdout:111669149699 2024-09-11T04:55:36.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:55:36.264 INFO:teuthology.orchestra.run.smithi031.stdout:38654705678 2024-09-11T04:55:36.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:55:36.569 INFO:teuthology.orchestra.run.smithi031.stdout:98784247811 2024-09-11T04:55:36.582 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247813 got 98784247811 for osd.0 2024-09-11T04:55:36.624 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-11T04:55:36.630 INFO:teuthology.orchestra.run.smithi031.stdout:38654705677 2024-09-11T04:55:36.638 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149699 got 1 for osd.3 2024-09-11T04:55:36.642 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705678 got 38654705677 for osd.2 2024-09-11T04:55:37.583 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:37.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:55:37.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:55:37.907 INFO:teuthology.orchestra.run.smithi031.stdout:98784247811 2024-09-11T04:55:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247813 got 98784247811 for osd.0 2024-09-11T04:55:37.982 INFO:teuthology.orchestra.run.smithi031.stdout:111669149699 2024-09-11T04:55:37.995 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149699 got 111669149699 for osd.3 2024-09-11T04:55:37.996 DEBUG:teuthology.parallel:result is None 2024-09-11T04:55:38.021 INFO:teuthology.orchestra.run.smithi031.stdout:38654705678 2024-09-11T04:55:38.034 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705678 got 38654705678 for osd.2 2024-09-11T04:55:38.034 DEBUG:teuthology.parallel:result is None 2024-09-11T04:55:38.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:39.231 INFO:teuthology.orchestra.run.smithi031.stdout:98784247813 2024-09-11T04:55:39.244 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247813 got 98784247813 for osd.0 2024-09-11T04:55:39.244 DEBUG:teuthology.parallel:result is None 2024-09-11T04:55:39.244 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2024-09-11T04:55:39.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:55:39.539 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:39.539 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:55:39.551 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-09-11T04:55:37.605081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188608,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":110092,"kb_used_data":1012,"kb_used_omap":11,"kb_used_meta":108980,"kb_avail":377377268,"statfs":{"total":386547056640,"available":386434322432,"internally_reserved":0,"allocated":1036288,"data_stored":475443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":111595817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.697994"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-11T04:55:32.929508+0000","last_change":"2024-09-11T04:55:32.929039+0000","last_active":"2024-09-11T04:55:32.929508+0000","last_peered":"2024-09-11T04:55:32.929508+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:55:32.929039+0000","last_became_peered":"2024-09-11T04:55:32.929039+0000","last_unstale":"2024-09-11T04:55:32.929508+0000","last_undegraded":"2024-09-11T04:55:32.929508+0000","last_fullsized":"2024-09-11T04:55:32.921148+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:50.426087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":43,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.943763+0000","last_change":"2024-09-11T04:55:32.944101+0000","last_active":"2024-09-11T04:55:32.943763+0000","last_peered":"2024-09-11T04:55:32.943763+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:55:32.932235+0000","last_became_peered":"2024-09-11T04:55:32.932235+0000","last_unstale":"2024-09-11T04:55:32.943763+0000","last_undegraded":"2024-09-11T04:55:32.925310+0000","last_fullsized":"2024-09-11T04:55:32.924905+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":56,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.939907+0000","last_change":"2024-09-11T04:55:32.940236+0000","last_active":"2024-09-11T04:55:32.939907+0000","last_peered":"2024-09-11T04:55:32.939907+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:55:32.933552+0000","last_became_peered":"2024-09-11T04:55:32.933552+0000","last_unstale":"2024-09-11T04:55:32.939907+0000","last_undegraded":"2024-09-11T04:55:32.920136+0000","last_fullsized":"2024-09-11T04:55:32.919825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":43,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.937492+0000","last_change":"2024-09-11T04:55:32.937810+0000","last_active":"2024-09-11T04:55:32.937492+0000","last_peered":"2024-09-11T04:55:32.937492+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:55:32.929693+0000","last_became_peered":"2024-09-11T04:55:32.929693+0000","last_unstale":"2024-09-11T04:55:32.937492+0000","last_undegraded":"2024-09-11T04:55:32.924293+0000","last_fullsized":"2024-09-11T04:55:32.923914+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":32,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.944408+0000","last_change":"2024-09-11T04:55:32.944700+0000","last_active":"2024-09-11T04:55:32.944408+0000","last_peered":"2024-09-11T04:55:32.944408+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:55:32.932766+0000","last_became_peered":"2024-09-11T04:55:32.932766+0000","last_unstale":"2024-09-11T04:55:32.944408+0000","last_undegraded":"2024-09-11T04:55:32.917722+0000","last_fullsized":"2024-09-11T04:55:32.917243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-11T04:55:32.934546+0000","last_change":"2024-09-11T04:55:32.934200+0000","last_active":"2024-09-11T04:55:32.934546+0000","last_peered":"2024-09-11T04:55:32.934546+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:55:32.934200+0000","last_became_peered":"2024-09-11T04:55:32.934200+0000","last_unstale":"2024-09-11T04:55:32.934546+0000","last_undegraded":"2024-09-11T04:55:32.934546+0000","last_fullsized":"2024-09-11T04:55:32.918822+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:46:05.656702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":45,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.942274+0000","last_change":"2024-09-11T04:55:32.942695+0000","last_active":"2024-09-11T04:55:32.942274+0000","last_peered":"2024-09-11T04:55:32.942274+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:55:32.931332+0000","last_became_peered":"2024-09-11T04:55:32.931332+0000","last_unstale":"2024-09-11T04:55:32.942274+0000","last_undegraded":"2024-09-11T04:55:32.923534+0000","last_fullsized":"2024-09-11T04:55:32.923137+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.939197+0000","last_change":"2024-09-11T04:55:32.939534+0000","last_active":"2024-09-11T04:55:32.939197+0000","last_peered":"2024-09-11T04:55:32.939197+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:55:32.930472+0000","last_became_peered":"2024-09-11T04:55:32.930472+0000","last_unstale":"2024-09-11T04:55:32.939197+0000","last_undegraded":"2024-09-11T04:55:32.925257+0000","last_fullsized":"2024-09-11T04:55:32.924845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":37,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.947073+0000","last_change":"2024-09-11T04:55:32.947363+0000","last_active":"2024-09-11T04:55:32.947073+0000","last_peered":"2024-09-11T04:55:32.947073+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:55:32.935391+0000","last_became_peered":"2024-09-11T04:55:32.935391+0000","last_unstale":"2024-09-11T04:55:32.947073+0000","last_undegraded":"2024-09-11T04:55:32.919426+0000","last_fullsized":"2024-09-11T04:55:32.919065+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":54,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.945298+0000","last_change":"2024-09-11T04:55:32.945524+0000","last_active":"2024-09-11T04:55:32.945298+0000","last_peered":"2024-09-11T04:55:32.945298+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:55:32.933998+0000","last_became_peered":"2024-09-11T04:55:32.933998+0000","last_unstale":"2024-09-11T04:55:32.945298+0000","last_undegraded":"2024-09-11T04:55:32.919878+0000","last_fullsized":"2024-09-11T04:55:32.919516+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":33,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.946087+0000","last_change":"2024-09-11T04:55:32.946384+0000","last_active":"2024-09-11T04:55:32.946087+0000","last_peered":"2024-09-11T04:55:32.946087+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:55:32.933398+0000","last_became_peered":"2024-09-11T04:55:32.933398+0000","last_unstale":"2024-09-11T04:55:32.946087+0000","last_undegraded":"2024-09-11T04:55:32.918010+0000","last_fullsized":"2024-09-11T04:55:32.917673+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":17,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.938529+0000","last_change":"2024-09-11T04:55:32.938529+0000","last_active":"2024-09-11T04:55:32.938529+0000","last_peered":"2024-09-11T04:55:32.938529+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:55:32.931253+0000","last_became_peered":"2024-09-11T04:55:32.931253+0000","last_unstale":"2024-09-11T04:55:32.938529+0000","last_undegraded":"2024-09-11T04:55:32.920335+0000","last_fullsized":"2024-09-11T04:55:32.919842+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":54,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.936544+0000","last_change":"2024-09-11T04:55:32.936954+0000","last_active":"2024-09-11T04:55:32.936544+0000","last_peered":"2024-09-11T04:55:32.936544+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:55:32.930786+0000","last_became_peered":"2024-09-11T04:55:32.930786+0000","last_unstale":"2024-09-11T04:55:32.936544+0000","last_undegraded":"2024-09-11T04:55:32.918841+0000","last_fullsized":"2024-09-11T04:55:32.918479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":32,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.941615+0000","last_change":"2024-09-11T04:55:32.941615+0000","last_active":"2024-09-11T04:55:32.941615+0000","last_peered":"2024-09-11T04:55:32.941615+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:55:32.931486+0000","last_became_peered":"2024-09-11T04:55:32.931486+0000","last_unstale":"2024-09-11T04:55:32.941615+0000","last_undegraded":"2024-09-11T04:55:32.917747+0000","last_fullsized":"2024-09-11T04:55:32.917411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":57,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.943051+0000","last_change":"2024-09-11T04:55:32.943389+0000","last_active":"2024-09-11T04:55:32.943051+0000","last_peered":"2024-09-11T04:55:32.943051+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:55:32.933937+0000","last_became_peered":"2024-09-11T04:55:32.933937+0000","last_unstale":"2024-09-11T04:55:32.943051+0000","last_undegraded":"2024-09-11T04:55:32.919524+0000","last_fullsized":"2024-09-11T04:55:32.919171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:32.940585+0000","last_change":"2024-09-11T04:55:32.940885+0000","last_active":"2024-09-11T04:55:32.940585+0000","last_peered":"2024-09-11T04:55:32.940585+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:55:32.929646+0000","last_became_peered":"2024-09-11T04:55:32.929646+0000","last_unstale":"2024-09-11T04:55:32.940585+0000","last_undegraded":"2024-09-11T04:55:32.924472+0000","last_fullsized":"2024-09-11T04:55:32.923989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":188608,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":233472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":270336,"data_stored":116951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":28176080},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":344064,"data_stored":190679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":50862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":270336,"data_stored":116951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:55:39.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:55:39.923 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:39.923 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":66,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726030531,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":188608,"bytes_used":112734208,"bytes_avail":386434322432,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-11T04:54:33.005366+0000","services":{}},"progress_events":{}} 2024-09-11T04:55:39.936 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '843fb9a6-9086-48d4-b349-4200e697289c', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 66, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726030531, 'num_in_osds': 4, 'osd_in_since': 1726030474, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 188608, 'bytes_used': 112734208, 'bytes_avail': 386434322432, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-11T04:54:33:002250+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-11T04:54:33.005366+0000', 'services': {}}, 'progress_events': {}} 2024-09-11T04:55:39.936 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2024-09-11T04:55:39.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T04:55:39.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T04:55:39.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T04:55:40.083 INFO:teuthology.orchestra.run.smithi031.stdout:111669149700 2024-09-11T04:55:40.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:55:40.091 INFO:teuthology.orchestra.run.smithi031.stdout:38654705680 2024-09-11T04:55:40.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:55:40.100 INFO:teuthology.orchestra.run.smithi031.stdout:98784247814 2024-09-11T04:55:40.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:40.422 INFO:teuthology.orchestra.run.smithi031.stdout:111669149699 2024-09-11T04:55:40.435 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149699 for osd.3 2024-09-11T04:55:40.448 INFO:teuthology.orchestra.run.smithi031.stdout:98784247813 2024-09-11T04:55:40.461 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-11T04:55:40.470 INFO:teuthology.orchestra.run.smithi031.stdout:38654705679 2024-09-11T04:55:40.483 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705680 got 38654705679 for osd.2 2024-09-11T04:55:41.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:55:41.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:41.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:55:41.765 INFO:teuthology.orchestra.run.smithi031.stdout:111669149699 2024-09-11T04:55:41.783 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149699 for osd.3 2024-09-11T04:55:41.800 INFO:teuthology.orchestra.run.smithi031.stdout:98784247813 2024-09-11T04:55:41.813 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-11T04:55:41.844 INFO:teuthology.orchestra.run.smithi031.stdout:38654705679 2024-09-11T04:55:41.857 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705680 got 38654705679 for osd.2 2024-09-11T04:55:42.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:55:42.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:55:42.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:55:43.120 INFO:teuthology.orchestra.run.smithi031.stdout:111669149701 2024-09-11T04:55:43.136 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149701 for osd.3 2024-09-11T04:55:43.136 DEBUG:teuthology.parallel:result is None 2024-09-11T04:55:43.151 INFO:teuthology.orchestra.run.smithi031.stdout:98784247815 2024-09-11T04:55:43.164 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247815 for osd.0 2024-09-11T04:55:43.164 DEBUG:teuthology.parallel:result is None 2024-09-11T04:55:43.188 INFO:teuthology.orchestra.run.smithi031.stdout:38654705680 2024-09-11T04:55:43.202 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705680 got 38654705680 for osd.2 2024-09-11T04:55:43.202 DEBUG:teuthology.parallel:result is None 2024-09-11T04:55:43.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:55:43.616 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:43.616 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":70,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726030531,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":188608,"bytes_used":112734208,"bytes_avail":386434322432,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-11T04:55:39.605619+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:55:43.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '843fb9a6-9086-48d4-b349-4200e697289c', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 70, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726030531, 'num_in_osds': 4, 'osd_in_since': 1726030474, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 188608, 'bytes_used': 112734208, 'bytes_avail': 386434322432, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-11T04:54:33:002250+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-09-11T04:55:39.605619+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '3': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-11T04:55:43.629 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2024-09-11T04:55:43.629 DEBUG:teuthology.orchestra.run.smithi031:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2024-09-11T04:55:43.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stderr:INFO: op_size has been rounded to 8192 2024-09-11T04:55:43.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:hints = 1 2024-09-11T04:55:43.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2024-09-11T04:55:43.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Object prefix: benchmark_data_smithi031_44851 2024-09-11T04:55:43.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:55:43.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-09-11T04:55:44.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 1 20 1347 1327 10.3665 10.3672 0.00886716 0.0149741 2024-09-11T04:55:45.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 2 20 2952 2932 11.4523 12.5391 0.012276 0.0136056 2024-09-11T04:55:46.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 3 20 4468 4448 11.5825 11.8438 0.0111257 0.0134504 2024-09-11T04:55:47.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 4 20 5977 5957 11.6339 11.7891 0.0173593 0.0133994 2024-09-11T04:55:48.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 5 20 7394 7374 11.521 11.0703 0.0165748 0.0135329 2024-09-11T04:55:49.682 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 6 20 8835 8815 11.477 11.2578 0.0103073 0.0135988 2024-09-11T04:55:50.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 7 20 10228 10208 11.392 10.8828 0.0118675 0.0136993 2024-09-11T04:55:51.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 8 20 11589 11569 11.297 10.6328 0.018649 0.013812 2024-09-11T04:55:52.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 9 20 12879 12859 11.1615 10.0781 0.0166184 0.0139845 2024-09-11T04:55:53.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:55:53.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 10 19 14166 14147 11.0515 10.0625 0.00962781 0.0141248 2024-09-11T04:55:54.198 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:55:54.198 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:55:54.222 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-11T04:55:53.609304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":79298752,"num_objects":9685,"num_object_clones":0,"num_object_copies":38740,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":9739,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":9685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9712,"num_write_kb":77578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9712,"ondisk_log_size":9712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":228076,"kb_used_data":102380,"kb_used_omap":11,"kb_used_meta":125620,"kb_avail":377259284,"statfs":{"total":386547056640,"available":386313506816,"internally_reserved":0,"allocated":104837120,"data_stored":104276275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":128635177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,7,10,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":79110144,"num_objects":9657,"num_object_clones":0,"num_object_copies":38628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":77256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.f","version":"27'669","reported_seq":698,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:55:50.908898+0000","last_change":"2024-09-11T04:55:43.715620+0000","last_active":"2024-09-11T04:55:50.908898+0000","last_peered":"2024-09-11T04:55:50.908898+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:55:32.929039+0000","last_became_peered":"2024-09-11T04:55:32.929039+0000","last_unstale":"2024-09-11T04:55:50.908898+0000","last_undegraded":"2024-09-11T04:55:43.693529+0000","last_fullsized":"2024-09-11T04:55:32.921148+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:50.426087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5480448,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":5352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'660","reported_seq":699,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.827934+0000","last_change":"2024-09-11T04:55:32.944101+0000","last_active":"2024-09-11T04:55:50.827934+0000","last_peered":"2024-09-11T04:55:50.827934+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:55:32.932235+0000","last_became_peered":"2024-09-11T04:55:32.932235+0000","last_unstale":"2024-09-11T04:55:50.827934+0000","last_undegraded":"2024-09-11T04:55:32.925310+0000","last_fullsized":"2024-09-11T04:55:32.924905+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5387424,"num_objects":658,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":662,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":660,"num_write_kb":5276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":656},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'445","reported_seq":498,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:48.559101+0000","last_change":"2024-09-11T04:55:32.940236+0000","last_active":"2024-09-11T04:55:48.559101+0000","last_peered":"2024-09-11T04:55:48.559101+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:55:32.933552+0000","last_became_peered":"2024-09-11T04:55:32.933552+0000","last_unstale":"2024-09-11T04:55:48.559101+0000","last_undegraded":"2024-09-11T04:55:32.920136+0000","last_fullsized":"2024-09-11T04:55:32.919825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3627600,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":445,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":3550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":442},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'623","reported_seq":662,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.804090+0000","last_change":"2024-09-11T04:55:32.937810+0000","last_active":"2024-09-11T04:55:50.804090+0000","last_peered":"2024-09-11T04:55:50.804090+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:55:32.929693+0000","last_became_peered":"2024-09-11T04:55:32.929693+0000","last_unstale":"2024-09-11T04:55:50.804090+0000","last_undegraded":"2024-09-11T04:55:32.924293+0000","last_fullsized":"2024-09-11T04:55:32.923914+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5084320,"num_objects":621,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":623,"num_write_kb":4973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":619},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'620","reported_seq":649,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.900007+0000","last_change":"2024-09-11T04:55:32.944700+0000","last_active":"2024-09-11T04:55:50.900007+0000","last_peered":"2024-09-11T04:55:50.900007+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:55:32.932766+0000","last_became_peered":"2024-09-11T04:55:32.932766+0000","last_unstale":"2024-09-11T04:55:50.900007+0000","last_undegraded":"2024-09-11T04:55:32.917722+0000","last_fullsized":"2024-09-11T04:55:32.917243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5067936,"num_objects":619,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":623,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":620,"num_write_kb":4957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":617},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'692","reported_seq":721,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:55:50.900101+0000","last_change":"2024-09-11T04:55:43.697029+0000","last_active":"2024-09-11T04:55:50.900101+0000","last_peered":"2024-09-11T04:55:50.900101+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:55:32.934200+0000","last_became_peered":"2024-09-11T04:55:32.934200+0000","last_unstale":"2024-09-11T04:55:50.900101+0000","last_undegraded":"2024-09-11T04:55:43.696802+0000","last_fullsized":"2024-09-11T04:55:32.918822+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:46:05.656702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5668864,"num_objects":692,"num_object_clones":0,"num_object_copies":2768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":5536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'625","reported_seq":664,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.833079+0000","last_change":"2024-09-11T04:55:32.942695+0000","last_active":"2024-09-11T04:55:50.833079+0000","last_peered":"2024-09-11T04:55:50.833079+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:55:32.931332+0000","last_became_peered":"2024-09-11T04:55:32.931332+0000","last_unstale":"2024-09-11T04:55:50.833079+0000","last_undegraded":"2024-09-11T04:55:32.923534+0000","last_fullsized":"2024-09-11T04:55:32.923137+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5084320,"num_objects":621,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":625,"num_write_kb":4980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":619},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'679","reported_seq":718,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.795446+0000","last_change":"2024-09-11T04:55:32.939534+0000","last_active":"2024-09-11T04:55:50.795446+0000","last_peered":"2024-09-11T04:55:50.795446+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:55:32.930472+0000","last_became_peered":"2024-09-11T04:55:32.930472+0000","last_unstale":"2024-09-11T04:55:50.795446+0000","last_undegraded":"2024-09-11T04:55:32.925257+0000","last_fullsized":"2024-09-11T04:55:32.924845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5534880,"num_objects":676,"num_object_clones":0,"num_object_copies":2704,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":680,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":679,"num_write_kb":5420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":674},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'699","reported_seq":727,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.882134+0000","last_change":"2024-09-11T04:55:32.947363+0000","last_active":"2024-09-11T04:55:50.882134+0000","last_peered":"2024-09-11T04:55:50.882134+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:55:32.935391+0000","last_became_peered":"2024-09-11T04:55:32.935391+0000","last_unstale":"2024-09-11T04:55:50.882134+0000","last_undegraded":"2024-09-11T04:55:32.919426+0000","last_fullsized":"2024-09-11T04:55:32.919065+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5679424,"num_objects":694,"num_object_clones":0,"num_object_copies":2776,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":700,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":699,"num_write_kb":5569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":691},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'449","reported_seq":500,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:48.600251+0000","last_change":"2024-09-11T04:55:32.945524+0000","last_active":"2024-09-11T04:55:48.600251+0000","last_peered":"2024-09-11T04:55:48.600251+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:55:32.933998+0000","last_became_peered":"2024-09-11T04:55:32.933998+0000","last_unstale":"2024-09-11T04:55:48.600251+0000","last_undegraded":"2024-09-11T04:55:32.919878+0000","last_fullsized":"2024-09-11T04:55:32.919516+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3675296,"num_objects":449,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":453,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":449,"num_write_kb":3590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":447},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'665","reported_seq":691,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.913251+0000","last_change":"2024-09-11T04:55:32.946384+0000","last_active":"2024-09-11T04:55:50.913251+0000","last_peered":"2024-09-11T04:55:50.913251+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:55:32.933398+0000","last_became_peered":"2024-09-11T04:55:32.933398+0000","last_unstale":"2024-09-11T04:55:50.913251+0000","last_undegraded":"2024-09-11T04:55:32.918010+0000","last_fullsized":"2024-09-11T04:55:32.917673+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5426928,"num_objects":663,"num_object_clones":0,"num_object_copies":2652,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":669,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":665,"num_write_kb":5308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":660},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'633","reported_seq":649,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.881811+0000","last_change":"2024-09-11T04:55:32.938529+0000","last_active":"2024-09-11T04:55:50.881811+0000","last_peered":"2024-09-11T04:55:50.881811+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:55:32.931253+0000","last_became_peered":"2024-09-11T04:55:32.931253+0000","last_unstale":"2024-09-11T04:55:50.881811+0000","last_undegraded":"2024-09-11T04:55:32.920335+0000","last_fullsized":"2024-09-11T04:55:32.919842+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5184080,"num_objects":633,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":635,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":633,"num_write_kb":5063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":632},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'489","reported_seq":542,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:48.594705+0000","last_change":"2024-09-11T04:55:32.936954+0000","last_active":"2024-09-11T04:55:48.594705+0000","last_peered":"2024-09-11T04:55:48.594705+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:55:32.930786+0000","last_became_peered":"2024-09-11T04:55:32.930786+0000","last_unstale":"2024-09-11T04:55:48.594705+0000","last_undegraded":"2024-09-11T04:55:32.918841+0000","last_fullsized":"2024-09-11T04:55:32.918479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4004432,"num_objects":489,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":489,"num_write_kb":3911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":488},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'651","reported_seq":677,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.888023+0000","last_change":"2024-09-11T04:55:32.941615+0000","last_active":"2024-09-11T04:55:50.888023+0000","last_peered":"2024-09-11T04:55:50.888023+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:55:32.931486+0000","last_became_peered":"2024-09-11T04:55:32.931486+0000","last_unstale":"2024-09-11T04:55:50.888023+0000","last_undegraded":"2024-09-11T04:55:32.917747+0000","last_fullsized":"2024-09-11T04:55:32.917411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5313696,"num_objects":649,"num_object_clones":0,"num_object_copies":2596,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":653,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":651,"num_write_kb":5204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":647},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'452","reported_seq":504,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:48.598142+0000","last_change":"2024-09-11T04:55:32.943389+0000","last_active":"2024-09-11T04:55:48.598142+0000","last_peered":"2024-09-11T04:55:48.598142+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:55:32.933937+0000","last_became_peered":"2024-09-11T04:55:32.933937+0000","last_unstale":"2024-09-11T04:55:48.598142+0000","last_undegraded":"2024-09-11T04:55:32.919524+0000","last_fullsized":"2024-09-11T04:55:32.919171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3691680,"num_objects":451,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":455,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":452,"num_write_kb":3613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":449},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'661","reported_seq":700,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-11T04:55:50.805558+0000","last_change":"2024-09-11T04:55:32.940885+0000","last_active":"2024-09-11T04:55:50.805558+0000","last_peered":"2024-09-11T04:55:50.805558+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:55:32.929646+0000","last_became_peered":"2024-09-11T04:55:32.929646+0000","last_unstale":"2024-09-11T04:55:50.805558+0000","last_undegraded":"2024-09-11T04:55:32.924472+0000","last_fullsized":"2024-09-11T04:55:32.923989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5387424,"num_objects":658,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":662,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":661,"num_write_kb":5276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":656},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":79298752,"num_objects":9685,"num_object_clones":0,"num_object_copies":38740,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":9739,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":9685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9712,"num_write_kb":77578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116166656,"data_stored":116166656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9712,"ondisk_log_size":9712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75400,"kb_used_data":39544,"kb_used_omap":4,"kb_used_meta":35835,"kb_avail":94296440,"statfs":{"total":96636764160,"available":96559554560,"internally_reserved":0,"allocated":40493056,"data_stored":40339671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":36695760},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60428,"kb_used_data":25084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94311412,"statfs":{"total":96636764160,"available":96574885888,"internally_reserved":0,"allocated":25686016,"data_stored":25532631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,9],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":50862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65076,"kb_used_data":37604,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94306764,"statfs":{"total":96636764160,"available":96570126336,"internally_reserved":0,"allocated":38506496,"data_stored":38353111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42835968,"data_stored":42835968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30031872,"data_stored":30031872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":43261952,"data_stored":43261952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:55:54.223 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2024-09-11T04:55:54.224 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:54.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2024-09-11T04:55:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:54.453 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:54.453 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:54.453 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:54.453 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_4", 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:54.454 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 1224592654, 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'3", 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'1", 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_6", 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 3101443614, 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'4", 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'2", 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:55:54.456 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(0)", 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:54.457 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:54.458 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:54.477 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-11T04:55:54.477 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2024-09-11T04:55:54.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2024-09-11T04:55:54.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 11 20 15235 15215 10.8053 8.34375 0.0224492 0.0144449 2024-09-11T04:55:54.736 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:55:54.760 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:54.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2024-09-11T04:55:55.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:55:55.146+0000 7f674e358640 0 --2- 172.21.15.31:0/2955838623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67400b55e0 0x7f67400151d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:55.192 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 1, 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 1, 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_7", 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 2358579165, 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'3", 2024-09-11T04:55:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "2(0)" 2024-09-11T04:55:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(1)", 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:55.198 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:55.213 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-11T04:55:55.213 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2024-09-11T04:55:55.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2024-09-11T04:55:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-11T04:55:55.503 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:55.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2024-09-11T04:55:55.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 12 20 16311 16291 10.6051 8.40625 0.0215737 0.0147183 2024-09-11T04:55:55.751 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:55.751 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:55.751 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:55.751 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:55.751 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:55.751 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:55.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:55.751 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_5", 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:55.752 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 4248945696, 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'3", 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:55.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_5", 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 4212789968, 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:55.754 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'4", 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:55.755 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(0)", 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:55.756 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:55.773 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-11T04:55:55.773 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2024-09-11T04:55:55.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2024-09-11T04:55:56.038 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:55:56.064 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:56.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2024-09-11T04:55:56.305 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:55:56.306 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_3", 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 459768081, 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:56.307 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'3", 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'1", 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_3", 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:56.308 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 480221105, 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'2", 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:56.309 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(1)", 2024-09-11T04:55:56.310 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:56.311 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:56.311 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:56.311 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:56.311 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:56.330 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-11T04:55:56.330 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2024-09-11T04:55:56.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2024-09-11T04:55:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:55:56.607 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2024-09-11T04:55:56.607 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:56.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2024-09-11T04:55:56.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 13 20 17357 17337 10.4179 8.17188 0.028797 0.014986 2024-09-11T04:55:56.847 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:56.847 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:56.847 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:56.847 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:56.847 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:56.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_6", 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 1708673474, 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:56.849 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'5", 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_9", 2024-09-11T04:55:56.850 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 831402034, 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'6", 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:56.851 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(2)", 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:56.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:56.853 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:56.853 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:56.853 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:56.872 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-11T04:55:56.873 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2024-09-11T04:55:56.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2024-09-11T04:55:57.155 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:55:57.178 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:57.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2024-09-11T04:55:57.416 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:57.416 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:57.416 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:57.416 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:55:57.417 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_8", 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 1901787523, 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'5", 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'3", 2024-09-11T04:55:57.418 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_1", 2024-09-11T04:55:57.419 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 3122379363, 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'4", 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:57.420 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:57.421 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:55:57.421 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:57.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:57.421 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:57.421 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:57.421 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:57.421 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:57.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:57.422 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(0)", 2024-09-11T04:55:57.422 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:57.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:57.422 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:57.422 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:57.422 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:57.442 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "16'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-11T04:55:57.442 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2024-09-11T04:55:57.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2024-09-11T04:55:57.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 14 20 18388 18368 10.2491 8.05469 0.0174756 0.0152307 2024-09-11T04:55:57.693 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:55:57.719 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:57.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2024-09-11T04:55:57.957 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:57.957 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:57.957 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:57.957 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:57.957 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:57.957 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:57.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:57.957 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 3, 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 3, 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_4", 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 873891716, 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'8", 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_7", 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 3996155460, 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'9", 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'6", 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:57.960 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_2", 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 1767667508, 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:57.961 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'7", 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:57.962 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(3)", 2024-09-11T04:55:57.963 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:57.963 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:57.963 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:57.963 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:57.963 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:57.978 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'9", 'have': "16'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-11T04:55:57.978 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2024-09-11T04:55:57.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2024-09-11T04:55:58.054 INFO:tasks.ceph.mon.a.smithi031.stderr:2024-09-11T04:55:58.054+0000 7f5da73cb640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 13 pgs recovery_unfound (PG_DAMAGED) 2024-09-11T04:55:58.239 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-11T04:55:58.265 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:58.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_1", 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 26281365, 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'1", 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "2(0)" 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_2", 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 3196340981, 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'2", 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: "2(0)" 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(3)", 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:58.513 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:58.529 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-11T04:55:58.529 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2024-09-11T04:55:58.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2024-09-11T04:55:58.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 15 20 19396 19376 10.0908 7.875 0.0173545 0.0154732 2024-09-11T04:55:58.808 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:55:58.825 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:58.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:59.078 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 3, 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 3, 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_8", 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 1861424966, 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'5", 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:59.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_6", 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 3910131350, 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:59.080 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'4", 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_4", 2024-09-11T04:55:59.081 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 2787839990, 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'3", 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: "2(3)" 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:59.082 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(2)", 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:59.083 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:59.102 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-11T04:55:59.102 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2024-09-11T04:55:59.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2024-09-11T04:55:59.366 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-11T04:55:59.393 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:59.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:59.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 1, 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 1, 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_7", 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 3761017527, 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:55:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'1", 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:55:59.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:55:59.632 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(0)", 2024-09-11T04:55:59.632 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:55:59.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:55:59.632 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:55:59.632 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:55:59.632 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:55:59.653 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-11T04:55:59.653 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2024-09-11T04:55:59.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2024-09-11T04:55:59.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 16 20 20403 20383 9.95175 7.86719 0.0226556 0.0156854 2024-09-11T04:55:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-11T04:55:59.954 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2024-09-11T04:55:59.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2024-09-11T04:56:00.240 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 1, 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 1, 2024-09-11T04:56:00.241 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_8", 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 2730327960, 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'1", 2024-09-11T04:56:00.242 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "2(1)" 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(0)", 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:00.243 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:00.244 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:56:00.244 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:56:00.263 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-11T04:56:00.263 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2024-09-11T04:56:00.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2024-09-11T04:56:00.517 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-11T04:56:00.536 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2024-09-11T04:56:00.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2024-09-11T04:56:00.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 17 20 21388 21368 9.81898 7.69531 0.0151184 0.0158998 2024-09-11T04:56:00.767 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:56:00.768 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_5", 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 2998549577, 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'4", 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'2", 2024-09-11T04:56:00.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_2", 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 4032093161, 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:00.770 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'3", 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'1", 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:56:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:56:00.772 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:56:00.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:00.772 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(0)", 2024-09-11T04:56:00.772 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:56:00.772 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:00.772 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:00.772 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:56:00.772 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:56:00.793 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-11T04:56:00.794 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2024-09-11T04:56:00.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2024-09-11T04:56:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:56:01.070 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2024-09-11T04:56:01.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:01.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_9", 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 1217469130, 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:01.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'3", 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'1", 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "2(1)" 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "new_9", 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:56:01.330 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 170713514, 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'2", 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: "2(1)" 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:01.331 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(0)", 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:56:01.332 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:56:01.351 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-11T04:56:01.351 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2024-09-11T04:56:01.351 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2024-09-11T04:56:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:56:01.509 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2024-09-11T04:56:01.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2024-09-11T04:56:01.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 18 20 22228 22208 9.63787 6.5625 0.0140306 0.0162007 2024-09-11T04:56:01.733 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:56:01.733 INFO:teuthology.orchestra.run.smithi031.stdout: "offset": { 2024-09-11T04:56:01.733 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "", 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": 0, 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 0, 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": -9223372036854775808, 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "num_missing": 2, 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "num_unfound": 2, 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "objects": [ 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existed_3", 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:01.734 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 943960171, 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'5", 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "0'0", 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:01.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": { 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "oid": "existing_1", 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "snapid": -2, 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "hash": 2885884443, 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 0, 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "namespace": "" 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "need": "19'4", 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "have": "16'1", 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "none", 2024-09-11T04:56:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: "clean_regions": "clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false", 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: "locations": [ 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: "2(2)" 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "NotRecovering", 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: "available_might_have_unfound": true, 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: "might_have_unfound": [ 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": "1(0)", 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "osd is down" 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:01.737 INFO:teuthology.orchestra.run.smithi031.stdout: "more": false 2024-09-11T04:56:01.738 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:56:01.765 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6736, 18446744073709544879)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-11T04:56:01.765 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2024-09-11T04:56:01.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2024-09-11T04:56:02.026 INFO:teuthology.orchestra.run.smithi031.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-11T04:56:02.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-11T04:56:02.295 INFO:teuthology.orchestra.run.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-11T04:56:02.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-11T04:56:02.564 INFO:teuthology.orchestra.run.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-11T04:56:02.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2024-09-11T04:56:02.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 19 20 23195 23175 9.5282 7.55469 0.0189089 0.0163874 2024-09-11T04:56:02.824 INFO:teuthology.orchestra.run.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-11T04:56:02.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T04:56:02.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T04:56:02.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T04:56:03.113 INFO:teuthology.orchestra.run.smithi031.stdout:98784247820 2024-09-11T04:56:03.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:56:03.120 INFO:teuthology.orchestra.run.smithi031.stdout:38654705685 2024-09-11T04:56:03.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:56:03.145 INFO:teuthology.orchestra.run.smithi031.stdout:111669149706 2024-09-11T04:56:03.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:56:03.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:56:03.685303+0000 min lat: 0.00434046 max lat: 0.213255 avg lat: 0.0167186 2024-09-11T04:56:03.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:56:03.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 20 20 23918 23898 9.33419 5.64844 0.0278571 0.0167186 2024-09-11T04:56:03.719 INFO:teuthology.orchestra.run.smithi031.stdout:98784247819 2024-09-11T04:56:03.743 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247820 got 98784247819 for osd.0 2024-09-11T04:56:03.771 INFO:teuthology.orchestra.run.smithi031.stdout:111669149705 2024-09-11T04:56:03.779 INFO:teuthology.orchestra.run.smithi031.stdout:38654705684 2024-09-11T04:56:03.796 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149705 for osd.3 2024-09-11T04:56:03.803 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705685 got 38654705684 for osd.2 2024-09-11T04:56:04.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 21 19 24969 24950 9.28103 8.21875 0.0179223 0.0168235 2024-09-11T04:56:04.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:56:04.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:56:04.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:56:05.438 INFO:teuthology.orchestra.run.smithi031.stdout:98784247820 2024-09-11T04:56:05.458 INFO:teuthology.orchestra.run.smithi031.stdout:38654705685 2024-09-11T04:56:05.461 INFO:teuthology.orchestra.run.smithi031.stdout:111669149706 2024-09-11T04:56:05.463 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247820 got 98784247820 for osd.0 2024-09-11T04:56:05.463 DEBUG:teuthology.parallel:result is None 2024-09-11T04:56:05.484 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705685 got 38654705685 for osd.2 2024-09-11T04:56:05.484 DEBUG:teuthology.parallel:result is None 2024-09-11T04:56:05.498 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149706 for osd.3 2024-09-11T04:56:05.498 DEBUG:teuthology.parallel:result is None 2024-09-11T04:56:05.498 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-11T04:56:05.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:05.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 22 20 25762 25742 9.14039 6.1875 0.020133 0.0170828 2024-09-11T04:56:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:06.150 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-11T04:56:05.612548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":193618112,"num_objects":23640,"num_object_clones":0,"num_object_copies":94560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23667,"num_write_kb":189218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23694,"ondisk_log_size":23694,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":584388,"kb_used_data":279616,"kb_used_omap":11,"kb_used_meta":304692,"kb_avail":376902972,"statfs":{"total":386547056640,"available":385948643328,"internally_reserved":0,"allocated":286326784,"data_stored":285765939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":312004905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,7,18,13,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":114319360,"num_objects":13955,"num_object_clones":0,"num_object_copies":55820,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":13901,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":13955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13955,"num_write_kb":111640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003243"},"pg_stats":[{"pgid":"1.f","version":"27'1478","reported_seq":1507,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.068874+0000","last_change":"2024-09-11T04:55:43.715620+0000","last_active":"2024-09-11T04:56:03.068874+0000","last_peered":"2024-09-11T04:56:03.068874+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:55:32.929039+0000","last_became_peered":"2024-09-11T04:55:32.929039+0000","last_unstale":"2024-09-11T04:56:03.068874+0000","last_undegraded":"2024-09-11T04:55:43.693529+0000","last_fullsized":"2024-09-11T04:55:32.921148+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1478,"log_dups_size":0,"ondisk_log_size":1478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:50.426087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12107776,"num_objects":1478,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":11824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1484","reported_seq":1534,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.034675+0000","last_change":"2024-09-11T04:55:54.808849+0000","last_active":"2024-09-11T04:56:03.034675+0000","last_peered":"2024-09-11T04:56:03.034675+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:55:32.932235+0000","last_became_peered":"2024-09-11T04:55:32.932235+0000","last_unstale":"2024-09-11T04:56:03.034675+0000","last_undegraded":"2024-09-11T04:55:32.925310+0000","last_fullsized":"2024-09-11T04:55:32.924905+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1484,"log_dups_size":0,"ondisk_log_size":1484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:54.986292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12121248,"num_objects":1480,"num_object_clones":0,"num_object_copies":5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1482,"num_write_kb":11852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1480}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1462","reported_seq":1523,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.636318+0000","last_change":"2024-09-11T04:55:55.528046+0000","last_active":"2024-09-11T04:56:03.636318+0000","last_peered":"2024-09-11T04:56:03.636318+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:55:32.933552+0000","last_became_peered":"2024-09-11T04:55:32.933552+0000","last_unstale":"2024-09-11T04:56:03.636318+0000","last_undegraded":"2024-09-11T04:55:32.920136+0000","last_fullsized":"2024-09-11T04:55:32.919825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:55.814100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11950672,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":11678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1446","reported_seq":1496,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.062064+0000","last_change":"2024-09-11T04:55:56.099801+0000","last_active":"2024-09-11T04:56:03.062064+0000","last_peered":"2024-09-11T04:56:03.062064+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:55:32.929693+0000","last_became_peered":"2024-09-11T04:55:32.929693+0000","last_unstale":"2024-09-11T04:56:03.062064+0000","last_undegraded":"2024-09-11T04:55:32.924293+0000","last_fullsized":"2024-09-11T04:55:32.923914+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1446,"log_dups_size":0,"ondisk_log_size":1446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:56.954114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11809952,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":11541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1442}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1389","reported_seq":1429,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.091764+0000","last_change":"2024-09-11T04:55:56.693717+0000","last_active":"2024-09-11T04:56:03.091764+0000","last_peered":"2024-09-11T04:56:03.091764+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:55:32.932766+0000","last_became_peered":"2024-09-11T04:55:32.932766+0000","last_unstale":"2024-09-11T04:56:03.091764+0000","last_undegraded":"2024-09-11T04:55:32.917722+0000","last_fullsized":"2024-09-11T04:55:32.917243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:57.358276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11351200,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":11093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1517","reported_seq":1546,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.093870+0000","last_change":"2024-09-11T04:55:43.697029+0000","last_active":"2024-09-11T04:56:03.093870+0000","last_peered":"2024-09-11T04:56:03.093870+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:55:32.934200+0000","last_became_peered":"2024-09-11T04:55:32.934200+0000","last_unstale":"2024-09-11T04:56:03.093870+0000","last_undegraded":"2024-09-11T04:55:43.696802+0000","last_fullsized":"2024-09-11T04:55:32.918822+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:46:05.656702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427264,"num_objects":1517,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":12136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1443","reported_seq":1493,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:02.965064+0000","last_change":"2024-09-11T04:55:57.217363+0000","last_active":"2024-09-11T04:56:02.965064+0000","last_peered":"2024-09-11T04:56:02.965064+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:55:32.931332+0000","last_became_peered":"2024-09-11T04:55:32.931332+0000","last_unstale":"2024-09-11T04:56:02.965064+0000","last_undegraded":"2024-09-11T04:55:32.923534+0000","last_fullsized":"2024-09-11T04:55:32.923137+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1443,"log_dups_size":0,"ondisk_log_size":1443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:57.909414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11768992,"num_objects":1437,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1441,"num_write_kb":11508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1437}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1531","reported_seq":1581,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.065904+0000","last_change":"2024-09-11T04:55:57.787237+0000","last_active":"2024-09-11T04:56:03.065904+0000","last_peered":"2024-09-11T04:56:03.065904+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:55:32.930472+0000","last_became_peered":"2024-09-11T04:55:32.930472+0000","last_unstale":"2024-09-11T04:56:03.065904+0000","last_undegraded":"2024-09-11T04:55:32.925257+0000","last_fullsized":"2024-09-11T04:55:32.924845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:58.896488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12498080,"num_objects":1526,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1529,"num_write_kb":12220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1587","reported_seq":1629,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.090686+0000","last_change":"2024-09-11T04:55:58.388957+0000","last_active":"2024-09-11T04:56:03.090686+0000","last_peered":"2024-09-11T04:56:03.090686+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:55:32.935391+0000","last_became_peered":"2024-09-11T04:55:32.935391+0000","last_unstale":"2024-09-11T04:56:03.090686+0000","last_undegraded":"2024-09-11T04:55:32.919426+0000","last_fullsized":"2024-09-11T04:55:32.919065+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:59.411446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12929344,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":12649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1481","reported_seq":1544,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.585890+0000","last_change":"2024-09-11T04:55:58.863725+0000","last_active":"2024-09-11T04:56:03.585890+0000","last_peered":"2024-09-11T04:56:03.585890+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:55:32.933998+0000","last_became_peered":"2024-09-11T04:55:32.933998+0000","last_unstale":"2024-09-11T04:56:03.585890+0000","last_undegraded":"2024-09-11T04:55:32.919878+0000","last_fullsized":"2024-09-11T04:55:32.919516+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:58.882437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12113056,"num_objects":1479,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":11830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1479}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1500","reported_seq":1542,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.101608+0000","last_change":"2024-09-11T04:55:59.486201+0000","last_active":"2024-09-11T04:56:03.101608+0000","last_peered":"2024-09-11T04:56:03.101608+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:55:32.933398+0000","last_became_peered":"2024-09-11T04:55:32.933398+0000","last_unstale":"2024-09-11T04:56:03.101608+0000","last_undegraded":"2024-09-11T04:55:32.918010+0000","last_fullsized":"2024-09-11T04:55:32.917673+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:00.381441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12242672,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1497,"num_write_kb":11964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1365","reported_seq":1388,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.095024+0000","last_change":"2024-09-11T04:55:59.969494+0000","last_active":"2024-09-11T04:56:03.095024+0000","last_peered":"2024-09-11T04:56:03.095024+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:55:32.931253+0000","last_became_peered":"2024-09-11T04:55:32.931253+0000","last_unstale":"2024-09-11T04:56:03.095024+0000","last_undegraded":"2024-09-11T04:55:32.920335+0000","last_fullsized":"2024-09-11T04:55:32.919842+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1365,"log_dups_size":0,"ondisk_log_size":1365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:01.355757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11172432,"num_objects":1364,"num_object_clones":0,"num_object_copies":5456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1364,"num_write_kb":10911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1364}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1519","reported_seq":1580,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.511172+0000","last_change":"2024-09-11T04:56:00.581036+0000","last_active":"2024-09-11T04:56:03.511172+0000","last_peered":"2024-09-11T04:56:03.511172+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:55:32.930786+0000","last_became_peered":"2024-09-11T04:55:32.930786+0000","last_unstale":"2024-09-11T04:56:03.511172+0000","last_undegraded":"2024-09-11T04:55:32.918841+0000","last_fullsized":"2024-09-11T04:55:32.918479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1519,"log_dups_size":0,"ondisk_log_size":1519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:00.876587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12434000,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1518,"num_write_kb":12143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1481","reported_seq":1520,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.067084+0000","last_change":"2024-09-11T04:56:01.167299+0000","last_active":"2024-09-11T04:56:03.067084+0000","last_peered":"2024-09-11T04:56:03.067084+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:55:32.931486+0000","last_became_peered":"2024-09-11T04:55:32.931486+0000","last_unstale":"2024-09-11T04:56:03.067084+0000","last_undegraded":"2024-09-11T04:55:32.917747+0000","last_fullsized":"2024-09-11T04:55:32.917411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:02.377834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12096672,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":11828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1519","reported_seq":1583,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.611415+0000","last_change":"2024-09-11T04:56:01.507219+0000","last_active":"2024-09-11T04:56:03.611415+0000","last_peered":"2024-09-11T04:56:03.611415+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:55:32.933937+0000","last_became_peered":"2024-09-11T04:55:32.933937+0000","last_unstale":"2024-09-11T04:56:03.611415+0000","last_undegraded":"2024-09-11T04:55:32.919524+0000","last_fullsized":"2024-09-11T04:55:32.919171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1519,"log_dups_size":0,"ondisk_log_size":1519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:01.854869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12416160,"num_objects":1516,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":12133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1492","reported_seq":1542,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.049827+0000","last_change":"2024-09-11T04:56:02.127836+0000","last_active":"2024-09-11T04:56:03.049827+0000","last_peered":"2024-09-11T04:56:03.049827+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:55:32.929646+0000","last_became_peered":"2024-09-11T04:55:32.929646+0000","last_unstale":"2024-09-11T04:56:03.049827+0000","last_undegraded":"2024-09-11T04:55:32.924472+0000","last_fullsized":"2024-09-11T04:55:32.923989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1492,"log_dups_size":0,"ondisk_log_size":1492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:02.970111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178592,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":11908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1487}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":193618112,"num_objects":23640,"num_object_clones":0,"num_object_copies":94560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23667,"num_write_kb":189218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290660352,"data_stored":290660352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23694,"ondisk_log_size":23694,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181796,"kb_used_data":91792,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94190044,"statfs":{"total":96636764160,"available":96450605056,"internally_reserved":0,"allocated":93995008,"data_stored":93841623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":92139216},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,10,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187348,"kb_used_data":93700,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94184492,"statfs":{"total":96636764160,"available":96444919808,"internally_reserved":0,"allocated":95948800,"data_stored":95795415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,7,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":50862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188072,"kb_used_data":93976,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94183768,"statfs":{"total":96636764160,"available":96444178432,"internally_reserved":0,"allocated":96231424,"data_stored":96078039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96333519},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":96350208,"data_stored":96350208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97845248,"data_stored":97845248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":96428032,"data_stored":96428032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:06.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:06.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 23 20 26689 26669 9.05784 7.24219 0.0162307 0.0172365 2024-09-11T04:56:06.746 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:06.747 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:06.773 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-11T04:56:05.612548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":193618112,"num_objects":23640,"num_object_clones":0,"num_object_copies":94560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23667,"num_write_kb":189218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23694,"ondisk_log_size":23694,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":584388,"kb_used_data":279616,"kb_used_omap":11,"kb_used_meta":304692,"kb_avail":376902972,"statfs":{"total":386547056640,"available":385948643328,"internally_reserved":0,"allocated":286326784,"data_stored":285765939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":312004905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,7,18,13,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":114319360,"num_objects":13955,"num_object_clones":0,"num_object_copies":55820,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":13901,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":13955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13955,"num_write_kb":111640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003243"},"pg_stats":[{"pgid":"1.f","version":"27'1478","reported_seq":1507,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.068874+0000","last_change":"2024-09-11T04:55:43.715620+0000","last_active":"2024-09-11T04:56:03.068874+0000","last_peered":"2024-09-11T04:56:03.068874+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:55:32.929039+0000","last_became_peered":"2024-09-11T04:55:32.929039+0000","last_unstale":"2024-09-11T04:56:03.068874+0000","last_undegraded":"2024-09-11T04:55:43.693529+0000","last_fullsized":"2024-09-11T04:55:32.921148+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1478,"log_dups_size":0,"ondisk_log_size":1478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:50.426087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12107776,"num_objects":1478,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":11824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1484","reported_seq":1534,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.034675+0000","last_change":"2024-09-11T04:55:54.808849+0000","last_active":"2024-09-11T04:56:03.034675+0000","last_peered":"2024-09-11T04:56:03.034675+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:55:32.932235+0000","last_became_peered":"2024-09-11T04:55:32.932235+0000","last_unstale":"2024-09-11T04:56:03.034675+0000","last_undegraded":"2024-09-11T04:55:32.925310+0000","last_fullsized":"2024-09-11T04:55:32.924905+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1484,"log_dups_size":0,"ondisk_log_size":1484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:54.986292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12121248,"num_objects":1480,"num_object_clones":0,"num_object_copies":5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1482,"num_write_kb":11852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1480}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1462","reported_seq":1523,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.636318+0000","last_change":"2024-09-11T04:55:55.528046+0000","last_active":"2024-09-11T04:56:03.636318+0000","last_peered":"2024-09-11T04:56:03.636318+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:55:32.933552+0000","last_became_peered":"2024-09-11T04:55:32.933552+0000","last_unstale":"2024-09-11T04:56:03.636318+0000","last_undegraded":"2024-09-11T04:55:32.920136+0000","last_fullsized":"2024-09-11T04:55:32.919825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:55.814100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11950672,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":11678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1446","reported_seq":1496,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.062064+0000","last_change":"2024-09-11T04:55:56.099801+0000","last_active":"2024-09-11T04:56:03.062064+0000","last_peered":"2024-09-11T04:56:03.062064+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:55:32.929693+0000","last_became_peered":"2024-09-11T04:55:32.929693+0000","last_unstale":"2024-09-11T04:56:03.062064+0000","last_undegraded":"2024-09-11T04:55:32.924293+0000","last_fullsized":"2024-09-11T04:55:32.923914+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1446,"log_dups_size":0,"ondisk_log_size":1446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:56.954114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11809952,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":11541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1442}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1389","reported_seq":1429,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.091764+0000","last_change":"2024-09-11T04:55:56.693717+0000","last_active":"2024-09-11T04:56:03.091764+0000","last_peered":"2024-09-11T04:56:03.091764+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:55:32.932766+0000","last_became_peered":"2024-09-11T04:55:32.932766+0000","last_unstale":"2024-09-11T04:56:03.091764+0000","last_undegraded":"2024-09-11T04:55:32.917722+0000","last_fullsized":"2024-09-11T04:55:32.917243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:57.358276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11351200,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":11093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1517","reported_seq":1546,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.093870+0000","last_change":"2024-09-11T04:55:43.697029+0000","last_active":"2024-09-11T04:56:03.093870+0000","last_peered":"2024-09-11T04:56:03.093870+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:55:32.934200+0000","last_became_peered":"2024-09-11T04:55:32.934200+0000","last_unstale":"2024-09-11T04:56:03.093870+0000","last_undegraded":"2024-09-11T04:55:43.696802+0000","last_fullsized":"2024-09-11T04:55:32.918822+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:46:05.656702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427264,"num_objects":1517,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":12136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1443","reported_seq":1493,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:02.965064+0000","last_change":"2024-09-11T04:55:57.217363+0000","last_active":"2024-09-11T04:56:02.965064+0000","last_peered":"2024-09-11T04:56:02.965064+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:55:32.931332+0000","last_became_peered":"2024-09-11T04:55:32.931332+0000","last_unstale":"2024-09-11T04:56:02.965064+0000","last_undegraded":"2024-09-11T04:55:32.923534+0000","last_fullsized":"2024-09-11T04:55:32.923137+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1443,"log_dups_size":0,"ondisk_log_size":1443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:57.909414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11768992,"num_objects":1437,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1441,"num_write_kb":11508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1437}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1531","reported_seq":1581,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.065904+0000","last_change":"2024-09-11T04:55:57.787237+0000","last_active":"2024-09-11T04:56:03.065904+0000","last_peered":"2024-09-11T04:56:03.065904+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:55:32.930472+0000","last_became_peered":"2024-09-11T04:55:32.930472+0000","last_unstale":"2024-09-11T04:56:03.065904+0000","last_undegraded":"2024-09-11T04:55:32.925257+0000","last_fullsized":"2024-09-11T04:55:32.924845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:58.896488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12498080,"num_objects":1526,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1529,"num_write_kb":12220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1587","reported_seq":1629,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.090686+0000","last_change":"2024-09-11T04:55:58.388957+0000","last_active":"2024-09-11T04:56:03.090686+0000","last_peered":"2024-09-11T04:56:03.090686+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:55:32.935391+0000","last_became_peered":"2024-09-11T04:55:32.935391+0000","last_unstale":"2024-09-11T04:56:03.090686+0000","last_undegraded":"2024-09-11T04:55:32.919426+0000","last_fullsized":"2024-09-11T04:55:32.919065+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:59.411446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12929344,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":12649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1481","reported_seq":1544,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.585890+0000","last_change":"2024-09-11T04:55:58.863725+0000","last_active":"2024-09-11T04:56:03.585890+0000","last_peered":"2024-09-11T04:56:03.585890+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:55:32.933998+0000","last_became_peered":"2024-09-11T04:55:32.933998+0000","last_unstale":"2024-09-11T04:56:03.585890+0000","last_undegraded":"2024-09-11T04:55:32.919878+0000","last_fullsized":"2024-09-11T04:55:32.919516+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:56:58.882437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12113056,"num_objects":1479,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":11830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1479}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1500","reported_seq":1542,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.101608+0000","last_change":"2024-09-11T04:55:59.486201+0000","last_active":"2024-09-11T04:56:03.101608+0000","last_peered":"2024-09-11T04:56:03.101608+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:55:32.933398+0000","last_became_peered":"2024-09-11T04:55:32.933398+0000","last_unstale":"2024-09-11T04:56:03.101608+0000","last_undegraded":"2024-09-11T04:55:32.918010+0000","last_fullsized":"2024-09-11T04:55:32.917673+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:00.381441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12242672,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1497,"num_write_kb":11964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1365","reported_seq":1388,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.095024+0000","last_change":"2024-09-11T04:55:59.969494+0000","last_active":"2024-09-11T04:56:03.095024+0000","last_peered":"2024-09-11T04:56:03.095024+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:55:32.931253+0000","last_became_peered":"2024-09-11T04:55:32.931253+0000","last_unstale":"2024-09-11T04:56:03.095024+0000","last_undegraded":"2024-09-11T04:55:32.920335+0000","last_fullsized":"2024-09-11T04:55:32.919842+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1365,"log_dups_size":0,"ondisk_log_size":1365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:01.355757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11172432,"num_objects":1364,"num_object_clones":0,"num_object_copies":5456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1364,"num_write_kb":10911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1364}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1519","reported_seq":1580,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.511172+0000","last_change":"2024-09-11T04:56:00.581036+0000","last_active":"2024-09-11T04:56:03.511172+0000","last_peered":"2024-09-11T04:56:03.511172+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:55:32.930786+0000","last_became_peered":"2024-09-11T04:55:32.930786+0000","last_unstale":"2024-09-11T04:56:03.511172+0000","last_undegraded":"2024-09-11T04:55:32.918841+0000","last_fullsized":"2024-09-11T04:55:32.918479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1519,"log_dups_size":0,"ondisk_log_size":1519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:00.876587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12434000,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1518,"num_write_kb":12143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1481","reported_seq":1520,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.067084+0000","last_change":"2024-09-11T04:56:01.167299+0000","last_active":"2024-09-11T04:56:03.067084+0000","last_peered":"2024-09-11T04:56:03.067084+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:55:32.931486+0000","last_became_peered":"2024-09-11T04:55:32.931486+0000","last_unstale":"2024-09-11T04:56:03.067084+0000","last_undegraded":"2024-09-11T04:55:32.917747+0000","last_fullsized":"2024-09-11T04:55:32.917411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:02.377834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12096672,"num_objects":1477,"num_object_clones":0,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":11828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1519","reported_seq":1583,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.611415+0000","last_change":"2024-09-11T04:56:01.507219+0000","last_active":"2024-09-11T04:56:03.611415+0000","last_peered":"2024-09-11T04:56:03.611415+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:55:32.933937+0000","last_became_peered":"2024-09-11T04:55:32.933937+0000","last_unstale":"2024-09-11T04:56:03.611415+0000","last_undegraded":"2024-09-11T04:55:32.919524+0000","last_fullsized":"2024-09-11T04:55:32.919171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1519,"log_dups_size":0,"ondisk_log_size":1519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:01.854869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12416160,"num_objects":1516,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":12133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1492","reported_seq":1542,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:03.049827+0000","last_change":"2024-09-11T04:56:02.127836+0000","last_active":"2024-09-11T04:56:03.049827+0000","last_peered":"2024-09-11T04:56:03.049827+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:55:32.929646+0000","last_became_peered":"2024-09-11T04:55:32.929646+0000","last_unstale":"2024-09-11T04:56:03.049827+0000","last_undegraded":"2024-09-11T04:55:32.924472+0000","last_fullsized":"2024-09-11T04:55:32.923989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":1492,"log_dups_size":0,"ondisk_log_size":1492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:02.970111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178592,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":11908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1487}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":193618112,"num_objects":23640,"num_object_clones":0,"num_object_copies":94560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23667,"num_write_kb":189218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290660352,"data_stored":290660352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23694,"ondisk_log_size":23694,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181796,"kb_used_data":91792,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94190044,"statfs":{"total":96636764160,"available":96450605056,"internally_reserved":0,"allocated":93995008,"data_stored":93841623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":92139216},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,10,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187348,"kb_used_data":93700,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94184492,"statfs":{"total":96636764160,"available":96444919808,"internally_reserved":0,"allocated":95948800,"data_stored":95795415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,7,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":50862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188072,"kb_used_data":93976,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94183768,"statfs":{"total":96636764160,"available":96444178432,"internally_reserved":0,"allocated":96231424,"data_stored":96078039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96333519},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":96350208,"data_stored":96350208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97845248,"data_stored":97845248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":96428032,"data_stored":96428032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:06.774 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-11T04:56:06.774 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2024-09-11T04:56:06.774 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2024-09-11T04:56:06.880 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2024-09-11T04:56:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:06.888 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2024-09-11T04:56:06.888 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2024-09-11T04:56:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2024-09-11T04:56:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.000 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2024-09-11T04:56:07.000 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2024-09-11T04:56:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2024-09-11T04:56:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.128 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2024-09-11T04:56:07.129 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2024-09-11T04:56:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2024-09-11T04:56:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.245 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2024-09-11T04:56:07.246 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2024-09-11T04:56:07.355 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2024-09-11T04:56:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.366 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2024-09-11T04:56:07.366 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2024-09-11T04:56:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2024-09-11T04:56:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.488 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2024-09-11T04:56:07.488 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2024-09-11T04:56:07.596 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2024-09-11T04:56:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.608 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2024-09-11T04:56:07.608 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2024-09-11T04:56:07.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 24 20 27608 27588 8.97956 7.17969 0.0104894 0.0173884 2024-09-11T04:56:07.717 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2024-09-11T04:56:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.726 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2024-09-11T04:56:07.727 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2024-09-11T04:56:07.830 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2024-09-11T04:56:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.838 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2024-09-11T04:56:07.839 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2024-09-11T04:56:07.936 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2024-09-11T04:56:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:07.945 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2024-09-11T04:56:07.945 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2024-09-11T04:56:08.058 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2024-09-11T04:56:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:08.068 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2024-09-11T04:56:08.068 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2024-09-11T04:56:08.172 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2024-09-11T04:56:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:08.177 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2024-09-11T04:56:08.178 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2024-09-11T04:56:08.293 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2024-09-11T04:56:08.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:08.305 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2024-09-11T04:56:08.305 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2024-09-11T04:56:08.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T04:56:08.592+0000 7f7dfcf85640 0 --2- 172.21.15.31:0/3824031977 >> [v2:172.21.15.31:3302/0,v1:172.21.15.31:6791/0] conn(0x55bc4f5d3d00 0x55bc4f5d6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T04:56:08.610 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2024-09-11T04:56:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:08.615 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2024-09-11T04:56:08.615 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2024-09-11T04:56:08.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 25 20 28542 28522 8.91223 7.29688 0.0131566 0.0175195 2024-09-11T04:56:08.723 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2024-09-11T04:56:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:08.731 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2024-09-11T04:56:08.732 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2024-09-11T04:56:08.839 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2024-09-11T04:56:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:08.852 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2024-09-11T04:56:08.852 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2024-09-11T04:56:08.950 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2024-09-11T04:56:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:08.963 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2024-09-11T04:56:08.963 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2024-09-11T04:56:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2024-09-11T04:56:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:09.083 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2024-09-11T04:56:09.083 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2024-09-11T04:56:09.189 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2024-09-11T04:56:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:09.197 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2024-09-11T04:56:09.197 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2024-09-11T04:56:09.307 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2024-09-11T04:56:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:09.318 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2024-09-11T04:56:09.319 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2024-09-11T04:56:09.429 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2024-09-11T04:56:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:09.436 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2024-09-11T04:56:09.436 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2024-09-11T04:56:09.547 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2024-09-11T04:56:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:09.556 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2024-09-11T04:56:09.556 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2024-09-11T04:56:09.645 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2024-09-11T04:56:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:09.657 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2024-09-11T04:56:09.658 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2024-09-11T04:56:09.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 26 20 29424 29404 8.83446 6.89062 0.0349371 0.0176743 2024-09-11T04:56:09.772 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2024-09-11T04:56:09.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:09.781 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2024-09-11T04:56:09.781 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2024-09-11T04:56:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2024-09-11T04:56:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:09.935 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2024-09-11T04:56:09.935 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2024-09-11T04:56:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2024-09-11T04:56:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:10.057 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2024-09-11T04:56:10.057 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2024-09-11T04:56:10.166 INFO:teuthology.orchestra.run.smithi031.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2024-09-11T04:56:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:56:10.176 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T04:56:10.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T04:56:10.179 INFO:tasks.ceph.osd.1:Started 2024-09-11T04:56:10.179 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T04:56:10.369 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:config diff 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:config diff get 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:config get 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:config help [] 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:config set ... 2024-09-11T04:56:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: invalid command 2024-09-11T04:56:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T04:56:10.373 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-11T04:56:10.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 27 20 30315 30295 8.76505 6.96094 0.0221241 0.0178168 2024-09-11T04:56:11.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 28 20 31287 31267 8.7232 7.59375 0.0184017 0.0178995 2024-09-11T04:56:11.894 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T04:56:11.893+0000 7f2004c1e640 -1 Falling back to public interface 2024-09-11T04:56:12.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 29 20 32237 32217 8.6783 7.42188 0.0212682 0.0179957 2024-09-11T04:56:13.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 30 20 33181 33161 8.63483 7.375 0.00982818 0.0180844 2024-09-11T04:56:14.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 31 20 34086 34066 8.58435 7.07031 0.0276335 0.0181911 2024-09-11T04:56:15.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T04:56:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-09-11T04:56:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-09-11T04:56:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-09-11T04:56:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-09-11T04:56:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-09-11T04:56:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-09-11T04:56:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-09-11T04:56:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator fragmentation block 2024-09-11T04:56:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator fragmentation histogram block [] [] 2024-09-11T04:56:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator score block 2024-09-11T04:56:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:config diff 2024-09-11T04:56:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:admin_socket: invalid command 2024-09-11T04:56:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T04:56:15.575 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-11T04:56:15.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 32 20 34841 34821 8.5004 5.89844 0.0291981 0.0183714 2024-09-11T04:56:15.732 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T04:56:15.732+0000 7f2004c1e640 -1 osd.1 19 log_to_monitors true 2024-09-11T04:56:16.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 33 20 35642 35622 8.43243 6.25781 0.0269257 0.0185177 2024-09-11T04:56:17.423 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T04:56:17.423+0000 7f1ff842b640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:56:17.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 34 20 36376 36356 8.35306 5.73438 0.0215071 0.0186965 2024-09-11T04:56:18.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 35 20 36864 36844 8.22331 3.8125 0.0207565 0.0187381 2024-09-11T04:56:19.686 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 36 20 36864 36844 7.99487 0 - 0.0187381 2024-09-11T04:56:20.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T04:56:20.688 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 37 20 36896 36876 7.78555 0.125 2.45895 0.0200432 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.ds1 31/30 ECSubWrite(tid=13063, reqid=client.5035.0:36911, at_version=31'2228, trim_to=0'0, roll_forward_to=31'2226))", 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.779616+0000", 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.02680985, 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026832040000000001, 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.807 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779616+0000", 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779616+0000", 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779617+0000", 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.91e-07 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779636+0000", 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9712999999999999e-05 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779643+0000", 2024-09-11T04:56:20.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1450000000000002e-06 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779649+0000", 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0839999999999999e-06 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.781539+0000", 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001890051 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.781549+0000", 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0113e-05 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.5s3 31/30 ECSubWrite(tid=13064, reqid=client.5035.0:36914, at_version=31'2325, trim_to=0'0, roll_forward_to=31'2323))", 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.779664+0000", 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.026761704000000001, 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026840875, 2024-09-11T04:56:20.811 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779664+0000", 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779664+0000", 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779665+0000", 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3299999999999998e-07 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779682+0000", 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7635e-05 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779691+0000", 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5799999999999992e-06 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779697+0000", 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.46e-06 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.786026+0000", 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063286740000000003 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.786037+0000", 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0803e-05 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.6s2 31/30 ECSubWrite(tid=22656, reqid=client.5035.0:36921, at_version=31'2334, trim_to=0'0, roll_forward_to=0'0))", 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.779700+0000", 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.026726025, 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026848461000000001, 2024-09-11T04:56:20.815 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779700+0000", 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779700+0000", 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779701+0000", 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7800000000000002e-07 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779722+0000", 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1781999999999999e-05 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779729+0000", 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.297e-06 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779736+0000", 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9990000000000002e-06 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.805727+0000", 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.025991223000000001 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.805734+0000", 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5039999999999996e-06 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.8s0 31/30 ECSubWrite(tid=13065, reqid=client.5035.0:36901, at_version=0'0, trim_to=0'0, roll_forward_to=31'2337))", 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.779708+0000", 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.026718151999999998, 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026882739999999999, 2024-09-11T04:56:20.819 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779708+0000", 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779708+0000", 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779708+0000", 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779724+0000", 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6121999999999999e-05 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779732+0000", 2024-09-11T04:56:20.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5560000000000002e-06 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779738+0000", 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7130000000000002e-06 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.782576+0000", 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028379859999999998 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.782586+0000", 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0454e-05 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.ds1 31/30 ECSubWrite(tid=13067, reqid=client.5035.0:36923, at_version=31'2229, trim_to=0'0, roll_forward_to=31'2226))", 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.779784+0000", 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.026642602000000001, 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026859166, 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.823 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779784+0000", 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779784+0000", 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779784+0000", 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.86e-07 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779810+0000", 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6597999999999998e-05 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779816+0000", 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.947e-06 2024-09-11T04:56:20.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779826+0000", 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0183999999999999e-05 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.800514+0000", 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020687938999999999 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.800523+0000", 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6619999999999994e-06 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.5s3 31/30 ECSubWrite(tid=13069, reqid=client.5035.0:36926, at_version=31'2326, trim_to=0'0, roll_forward_to=31'2323))", 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.788995+0000", 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.017430681999999999, 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017702437000000001, 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.827 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.788995+0000", 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.788995+0000", 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.788997+0000", 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0640000000000001e-06 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.789024+0000", 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7840000000000001e-05 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.789034+0000", 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6330000000000008e-06 2024-09-11T04:56:20.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.789043+0000", 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3959999999999995e-06 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.804661+0000", 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015617115000000001 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.804670+0000", 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2890000000000003e-06 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.3s0 31/30 ECSubWrite(tid=16092, reqid=client.5035.0:36905, at_version=31'2371, trim_to=0'0, roll_forward_to=0'0))", 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.724218+0000", 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.082208234000000005, 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.082526422000000002, 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.724218+0000", 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.724218+0000", 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.724219+0000", 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0210000000000001e-06 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.724253+0000", 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4081000000000001e-05 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.724265+0000", 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2036e-05 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.724275+0000", 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6090000000000007e-06 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.727530+0000", 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032557939999999998 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.727540+0000", 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8490000000000001e-06 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.3s0 31/30 ECSubWrite(tid=16100, reqid=client.5035.0:36928, at_version=31'2372, trim_to=0'0, roll_forward_to=31'2370))", 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.792287+0000", 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.014138809, 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014502803999999999, 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.792287+0000", 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.792287+0000", 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.792288+0000", 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3600000000000002e-07 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.792309+0000", 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0778e-05 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.792320+0000", 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1124999999999999e-05 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.792329+0000", 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.799e-06 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.0s0 31/30 ECSubWrite(tid=16101, reqid=client.5035.0:36927, at_version=31'2291, trim_to=0'0, roll_forward_to=31'2290))", 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.793023+0000", 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.013402860000000001, 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013807886, 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.838 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.793023+0000", 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.793023+0000", 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.793024+0000", 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6400000000000002e-07 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.793042+0000", 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8002000000000001e-05 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.793052+0000", 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.047e-05 2024-09-11T04:56:20.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.793060+0000", 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8569999999999995e-06 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.798803+0000", 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005743176 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.798813+0000", 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3610000000000007e-06 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.bs0 31/30 ECSubWrite(tid=16102, reqid=client.5035.0:36920, at_version=0'0, trim_to=0'0, roll_forward_to=31'2325))", 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.795903+0000", 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.010523271000000001, 2024-09-11T04:56:20.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010973675, 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.795903+0000", 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.795903+0000", 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.795904+0000", 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1060000000000001e-06 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.795919+0000", 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4805e-05 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.795928+0000", 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3610000000000007e-06 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.795936+0000", 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8290000000000008e-06 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.803102+0000", 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0071660300000000003 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.803111+0000", 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4029999999999996e-06 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.8s0 31/30 ECSubWrite(tid=13070, reqid=client.5035.0:36931, at_version=31'2338, trim_to=0'0, roll_forward_to=31'2337))", 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.798149+0000", 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0082772990000000001, 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0087739140000000007, 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.798149+0000", 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.798149+0000", 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.798150+0000", 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.068e-06 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.798172+0000", 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2357999999999998e-05 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.798183+0000", 2024-09-11T04:56:20.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0521e-05 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.798193+0000", 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9580000000000002e-06 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.805280+0000", 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0070875110000000003 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.805290+0000", 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9140000000000003e-06 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.bs0 31/30 ECSubWrite(tid=16103, reqid=client.5035.0:36932, at_version=31'2326, trim_to=0'0, roll_forward_to=31'2325))", 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.799936+0000", 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0064901159999999998, 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0070321350000000001, 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.850 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.799936+0000", 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.799936+0000", 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.799937+0000", 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1900000000000002e-07 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.799958+0000", 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1347000000000001e-05 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.799968+0000", 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.611e-06 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.799974+0000", 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.759e-06 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.805471+0000", 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054965259999999998 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.805479+0000", 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1720000000000007e-06 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDECSubOpWrite(1.5s3 31/30 ECSubWrite(tid=13061, reqid=client.5035.0:36910, at_version=31'2324, trim_to=0'0, roll_forward_to=31'2323))", 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-11T04:56:20.738251+0000", 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.068175056999999997, 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.068763736000000006, 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.738251+0000", 2024-09-11T04:56:20.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.738251+0000", 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.738252+0000", 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.139e-06 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.738287+0000", 2024-09-11T04:56:20.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4609e-05 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.738301+0000", 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3889e-05 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779525+0000", 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.041224201000000002 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-11T04:56:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779949+0000", 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00042397499999999999 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_started", 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-11T04:56:20.779959+0000", 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5789999999999993e-06 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 13 2024-09-11T04:56:20.858 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T04:56:20.859 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2024-09-11T04:56:20.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:56:21.540 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:21.540 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T04:56:20.209615+0000","last_up_change":"2024-09-11T04:56:18.205049+0000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-11T04:54:55.691815+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":30,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":448159694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":448159694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":448159694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":448159694}]},"public_addr":"172.21.15.31:6800/448159694","cluster_addr":"172.21.15.31:6801/448159694","heartbeat_back_addr":"172.21.15.31:6803/448159694","heartbeat_front_addr":"172.21.15.31:6802/448159694","state":["exists","up"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":29,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":815718084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":815718084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":815718084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":815718084}]},"public_addr":"172.21.15.31:6812/815718084","cluster_addr":"172.21.15.31:6813/815718084","heartbeat_back_addr":"172.21.15.31:6815/815718084","heartbeat_front_addr":"172.21.15.31:6814/815718084","state":["exists","up"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2027023681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2027023681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2027023681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2027023681}]},"public_addr":"172.21.15.31:6808/2027023681","cluster_addr":"172.21.15.31:6809/2027023681","heartbeat_back_addr":"172.21.15.31:6811/2027023681","heartbeat_front_addr":"172.21.15.31:6810/2027023681","state":["exists","up"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":30,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2434669726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2434669726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":2434669726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2434669726}]},"public_addr":"172.21.15.31:6804/2434669726","cluster_addr":"172.21.15.31:6805/2434669726","heartbeat_back_addr":"172.21.15.31:6807/2434669726","heartbeat_front_addr":"172.21.15.31:6806/2434669726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.833079+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.663344+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.858324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:41.136761+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[2147483647,0,2,3]},{"pgid":"1.1","osds":[3,2147483647,0,2]},{"pgid":"1.2","osds":[0,3,2147483647,2]},{"pgid":"1.3","osds":[2147483647,0,2,3]},{"pgid":"1.4","osds":[3,0,2,2147483647]},{"pgid":"1.5","osds":[2,0,3,2147483647]},{"pgid":"1.6","osds":[3,0,2147483647,2]},{"pgid":"1.7","osds":[2147483647,3,2,0]},{"pgid":"1.8","osds":[2147483647,2,3,0]},{"pgid":"1.9","osds":[2147483647,3,2,0]},{"pgid":"1.a","osds":[2147483647,2,3,0]},{"pgid":"1.b","osds":[2147483647,0,2,3]},{"pgid":"1.c","osds":[3,0,2,2147483647]},{"pgid":"1.d","osds":[2,2147483647,3,0]},{"pgid":"1.e","osds":[2147483647,0,3,2]},{"pgid":"1.f","osds":[3,0,2147483647,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:56:21.571 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2024-09-11T04:56:21.571 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-11T04:56:21.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:21.686 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 38 20 37291 37271 7.66187 3.08594 0.0188835 0.0203775 2024-09-11T04:56:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:22.209 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:22.235 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-11T04:56:21.617343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298909888,"num_objects":36493,"num_object_clones":0,"num_object_copies":145972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36520,"num_write_kb":292042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36547,"ondisk_log_size":36547,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":915540,"kb_used_data":442300,"kb_used_omap":14,"kb_used_meta":473137,"kb_avail":376571820,"statfs":{"total":386547056640,"available":385609543680,"internally_reserved":0,"allocated":452915200,"data_stored":452249550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":484492854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,5,3,10,1,1,0,10,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15491072,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1891,"num_write_kb":15128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.410984"},"pg_stats":[{"pgid":"1.f","version":"31'2315","reported_seq":2356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.885390+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:20.885390+0000","last_peered":"2024-09-11T04:56:20.885390+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:20.885390+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18964480,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":18520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2293","reported_seq":2353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.848133+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:20.848133+0000","last_peered":"2024-09-11T04:56:20.848133+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:20.848133+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18748576,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":18324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2224","reported_seq":2288,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-11T04:56:18.211765+0000","last_change":"2024-09-11T04:56:18.232581+0000","last_active":"2024-09-11T04:56:18.232581+0000","last_peered":"2024-09-11T04:56:18.211765+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:55:32.933552+0000","last_became_peered":"2024-09-11T04:55:32.933552+0000","last_unstale":"2024-09-11T04:56:18.211765+0000","last_undegraded":"2024-09-11T04:55:32.920136+0000","last_fullsized":"2024-09-11T04:55:32.919825+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18192976,"num_objects":2221,"num_object_clones":0,"num_object_copies":8884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2223,"num_write_kb":17774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2291","reported_seq":2353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.753538+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:20.753538+0000","last_peered":"2024-09-11T04:56:20.753538+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:20.753538+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2291,"log_dups_size":0,"ondisk_log_size":2291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18732192,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2263","reported_seq":2316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.881676+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:20.881676+0000","last_peered":"2024-09-11T04:56:20.881676+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:20.881676+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18511008,"num_objects":2260,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":18085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2332","reported_seq":2372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.925689+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:20.925689+0000","last_peered":"2024-09-11T04:56:20.925689+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:20.925689+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2332,"num_write_kb":18656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2332","reported_seq":2394,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.763669+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:20.763669+0000","last_peered":"2024-09-11T04:56:20.763669+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:20.763669+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19051680,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":18620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2372","reported_seq":2433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.826844+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:20.826844+0000","last_peered":"2024-09-11T04:56:20.826844+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:20.826844+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2372,"log_dups_size":0,"ondisk_log_size":2372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387552,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":18948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2415","reported_seq":2470,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:21.025875+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:21.025875+0000","last_peered":"2024-09-11T04:56:21.025875+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:21.025875+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19712320,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":19273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2322","reported_seq":2388,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-11T04:56:18.220809+0000","last_change":"2024-09-11T04:56:18.231170+0000","last_active":"2024-09-11T04:56:18.231170+0000","last_peered":"2024-09-11T04:56:18.220809+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:55:32.933998+0000","last_became_peered":"2024-09-11T04:55:32.933998+0000","last_unstale":"2024-09-11T04:56:18.220809+0000","last_undegraded":"2024-09-11T04:55:32.919878+0000","last_fullsized":"2024-09-11T04:55:32.919516+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2341","reported_seq":2393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.971079+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:20.971079+0000","last_peered":"2024-09-11T04:56:20.971079+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:20.971079+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2341,"log_dups_size":0,"ondisk_log_size":2341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19132144,"num_objects":2336,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2244","reported_seq":2294,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.965547+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:20.965547+0000","last_peered":"2024-09-11T04:56:20.965547+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:20.965547+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18373200,"num_objects":2243,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":17943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'2079","reported_seq":2140,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:13.642870+0000","last_change":"2024-09-11T04:56:00.581036+0000","last_active":"2024-09-11T04:56:13.642870+0000","last_peered":"2024-09-11T04:56:13.642870+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:55:32.930786+0000","last_became_peered":"2024-09-11T04:55:32.930786+0000","last_unstale":"2024-09-11T04:56:13.642870+0000","last_undegraded":"2024-09-11T04:55:32.918841+0000","last_fullsized":"2024-09-11T04:55:32.918479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2079,"log_dups_size":0,"ondisk_log_size":2079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:00.876587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17021520,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":16623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2306","reported_seq":2355,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.926138+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:20.926138+0000","last_peered":"2024-09-11T04:56:20.926138+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:20.926138+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18855072,"num_objects":2302,"num_object_clones":0,"num_object_copies":9208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":18428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'2092","reported_seq":2157,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:13.580150+0000","last_change":"2024-09-11T04:56:01.507219+0000","last_active":"2024-09-11T04:56:13.580150+0000","last_peered":"2024-09-11T04:56:13.580150+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:55:32.933937+0000","last_became_peered":"2024-09-11T04:55:32.933937+0000","last_unstale":"2024-09-11T04:56:13.580150+0000","last_undegraded":"2024-09-11T04:55:32.919524+0000","last_fullsized":"2024-09-11T04:55:32.919171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2092,"log_dups_size":0,"ondisk_log_size":2092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:01.854869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17110176,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":16717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2326","reported_seq":2388,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.786312+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:20.786312+0000","last_peered":"2024-09-11T04:56:20.786312+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:20.786312+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2326,"log_dups_size":0,"ondisk_log_size":2326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19010720,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":298909888,"num_objects":36493,"num_object_clones":0,"num_object_copies":145972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36520,"num_write_kb":292042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455159808,"data_stored":455159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36547,"ondisk_log_size":36547,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296000,"kb_used_data":147688,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94075840,"statfs":{"total":96636764160,"available":96333660160,"internally_reserved":0,"allocated":151232512,"data_stored":151061702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,3,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281692,"kb_used_data":146380,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94090148,"statfs":{"total":96636764160,"available":96348311552,"internally_reserved":0,"allocated":149893120,"data_stored":149735363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":45499,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":569344,"data_stored":398534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46591698},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291764,"kb_used_data":147676,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94080076,"statfs":{"total":96636764160,"available":96337997824,"internally_reserved":0,"allocated":151220224,"data_stored":151053951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,10,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151388160,"data_stored":151388160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1150976,"data_stored":1150976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":150966272,"data_stored":150966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151654400,"data_stored":151654400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:22.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:22.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 39 20 37670 37650 7.54114 2.96094 0.0466761 0.0206968 2024-09-11T04:56:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:22.843 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:22.862 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-11T04:56:21.617343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298909888,"num_objects":36493,"num_object_clones":0,"num_object_copies":145972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36520,"num_write_kb":292042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36547,"ondisk_log_size":36547,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":915540,"kb_used_data":442300,"kb_used_omap":14,"kb_used_meta":473137,"kb_avail":376571820,"statfs":{"total":386547056640,"available":385609543680,"internally_reserved":0,"allocated":452915200,"data_stored":452249550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":484492854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,5,3,10,1,1,0,10,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15491072,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1891,"num_write_kb":15128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.410984"},"pg_stats":[{"pgid":"1.f","version":"31'2315","reported_seq":2356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.885390+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:20.885390+0000","last_peered":"2024-09-11T04:56:20.885390+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:20.885390+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18964480,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":18520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2293","reported_seq":2353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.848133+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:20.848133+0000","last_peered":"2024-09-11T04:56:20.848133+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:20.848133+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18748576,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":18324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2224","reported_seq":2288,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-11T04:56:18.211765+0000","last_change":"2024-09-11T04:56:18.232581+0000","last_active":"2024-09-11T04:56:18.232581+0000","last_peered":"2024-09-11T04:56:18.211765+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:55:32.933552+0000","last_became_peered":"2024-09-11T04:55:32.933552+0000","last_unstale":"2024-09-11T04:56:18.211765+0000","last_undegraded":"2024-09-11T04:55:32.920136+0000","last_fullsized":"2024-09-11T04:55:32.919825+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18192976,"num_objects":2221,"num_object_clones":0,"num_object_copies":8884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2223,"num_write_kb":17774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2291","reported_seq":2353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.753538+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:20.753538+0000","last_peered":"2024-09-11T04:56:20.753538+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:20.753538+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2291,"log_dups_size":0,"ondisk_log_size":2291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18732192,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2263","reported_seq":2316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.881676+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:20.881676+0000","last_peered":"2024-09-11T04:56:20.881676+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:20.881676+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18511008,"num_objects":2260,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":18085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2332","reported_seq":2372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.925689+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:20.925689+0000","last_peered":"2024-09-11T04:56:20.925689+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:20.925689+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2332,"num_write_kb":18656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2332","reported_seq":2394,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.763669+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:20.763669+0000","last_peered":"2024-09-11T04:56:20.763669+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:20.763669+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19051680,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":18620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2372","reported_seq":2433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.826844+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:20.826844+0000","last_peered":"2024-09-11T04:56:20.826844+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:20.826844+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2372,"log_dups_size":0,"ondisk_log_size":2372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387552,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":18948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2415","reported_seq":2470,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:21.025875+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:21.025875+0000","last_peered":"2024-09-11T04:56:21.025875+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:21.025875+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19712320,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":19273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2322","reported_seq":2388,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-11T04:56:18.220809+0000","last_change":"2024-09-11T04:56:18.231170+0000","last_active":"2024-09-11T04:56:18.231170+0000","last_peered":"2024-09-11T04:56:18.220809+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:55:32.933998+0000","last_became_peered":"2024-09-11T04:55:32.933998+0000","last_unstale":"2024-09-11T04:56:18.220809+0000","last_undegraded":"2024-09-11T04:55:32.919878+0000","last_fullsized":"2024-09-11T04:55:32.919516+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2341","reported_seq":2393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.971079+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:20.971079+0000","last_peered":"2024-09-11T04:56:20.971079+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:20.971079+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2341,"log_dups_size":0,"ondisk_log_size":2341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19132144,"num_objects":2336,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2244","reported_seq":2294,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.965547+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:20.965547+0000","last_peered":"2024-09-11T04:56:20.965547+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:20.965547+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18373200,"num_objects":2243,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":17943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'2079","reported_seq":2140,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:13.642870+0000","last_change":"2024-09-11T04:56:00.581036+0000","last_active":"2024-09-11T04:56:13.642870+0000","last_peered":"2024-09-11T04:56:13.642870+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:55:32.930786+0000","last_became_peered":"2024-09-11T04:55:32.930786+0000","last_unstale":"2024-09-11T04:56:13.642870+0000","last_undegraded":"2024-09-11T04:55:32.918841+0000","last_fullsized":"2024-09-11T04:55:32.918479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2079,"log_dups_size":0,"ondisk_log_size":2079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:00.876587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17021520,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":16623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2306","reported_seq":2355,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.926138+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:20.926138+0000","last_peered":"2024-09-11T04:56:20.926138+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:20.926138+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18855072,"num_objects":2302,"num_object_clones":0,"num_object_copies":9208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":18428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'2092","reported_seq":2157,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:13.580150+0000","last_change":"2024-09-11T04:56:01.507219+0000","last_active":"2024-09-11T04:56:13.580150+0000","last_peered":"2024-09-11T04:56:13.580150+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:55:32.933937+0000","last_became_peered":"2024-09-11T04:55:32.933937+0000","last_unstale":"2024-09-11T04:56:13.580150+0000","last_undegraded":"2024-09-11T04:55:32.919524+0000","last_fullsized":"2024-09-11T04:55:32.919171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2092,"log_dups_size":0,"ondisk_log_size":2092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:01.854869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17110176,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":16717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2326","reported_seq":2388,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.786312+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:20.786312+0000","last_peered":"2024-09-11T04:56:20.786312+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:20.786312+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2326,"log_dups_size":0,"ondisk_log_size":2326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19010720,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":298909888,"num_objects":36493,"num_object_clones":0,"num_object_copies":145972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36520,"num_write_kb":292042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455159808,"data_stored":455159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36547,"ondisk_log_size":36547,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296000,"kb_used_data":147688,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94075840,"statfs":{"total":96636764160,"available":96333660160,"internally_reserved":0,"allocated":151232512,"data_stored":151061702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,3,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281692,"kb_used_data":146380,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94090148,"statfs":{"total":96636764160,"available":96348311552,"internally_reserved":0,"allocated":149893120,"data_stored":149735363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":45499,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":569344,"data_stored":398534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46591698},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291764,"kb_used_data":147676,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94080076,"statfs":{"total":96636764160,"available":96337997824,"internally_reserved":0,"allocated":151220224,"data_stored":151053951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,10,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151388160,"data_stored":151388160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1150976,"data_stored":1150976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":150966272,"data_stored":150966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151654400,"data_stored":151654400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:22.863 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:23.518 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-11T04:56:21.617343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298909888,"num_objects":36493,"num_object_clones":0,"num_object_copies":145972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36520,"num_write_kb":292042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36547,"ondisk_log_size":36547,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":915540,"kb_used_data":442300,"kb_used_omap":14,"kb_used_meta":473137,"kb_avail":376571820,"statfs":{"total":386547056640,"available":385609543680,"internally_reserved":0,"allocated":452915200,"data_stored":452249550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":484492854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,5,3,10,1,1,0,10,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15491072,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1891,"num_write_kb":15128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.410984"},"pg_stats":[{"pgid":"1.f","version":"31'2315","reported_seq":2356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.885390+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:20.885390+0000","last_peered":"2024-09-11T04:56:20.885390+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:20.885390+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18964480,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":18520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2293","reported_seq":2353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.848133+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:20.848133+0000","last_peered":"2024-09-11T04:56:20.848133+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:20.848133+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18748576,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":18324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2224","reported_seq":2288,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-11T04:56:18.211765+0000","last_change":"2024-09-11T04:56:18.232581+0000","last_active":"2024-09-11T04:56:18.232581+0000","last_peered":"2024-09-11T04:56:18.211765+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:55:32.933552+0000","last_became_peered":"2024-09-11T04:55:32.933552+0000","last_unstale":"2024-09-11T04:56:18.211765+0000","last_undegraded":"2024-09-11T04:55:32.920136+0000","last_fullsized":"2024-09-11T04:55:32.919825+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18192976,"num_objects":2221,"num_object_clones":0,"num_object_copies":8884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2223,"num_write_kb":17774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2291","reported_seq":2353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.753538+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:20.753538+0000","last_peered":"2024-09-11T04:56:20.753538+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:20.753538+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2291,"log_dups_size":0,"ondisk_log_size":2291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18732192,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2263","reported_seq":2316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.881676+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:20.881676+0000","last_peered":"2024-09-11T04:56:20.881676+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:20.881676+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18511008,"num_objects":2260,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":18085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2332","reported_seq":2372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.925689+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:20.925689+0000","last_peered":"2024-09-11T04:56:20.925689+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:20.925689+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2332,"num_write_kb":18656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2332","reported_seq":2394,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.763669+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:20.763669+0000","last_peered":"2024-09-11T04:56:20.763669+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:20.763669+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19051680,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":18620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2372","reported_seq":2433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.826844+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:20.826844+0000","last_peered":"2024-09-11T04:56:20.826844+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:20.826844+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2372,"log_dups_size":0,"ondisk_log_size":2372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387552,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":18948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2415","reported_seq":2470,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:21.025875+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:21.025875+0000","last_peered":"2024-09-11T04:56:21.025875+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:21.025875+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19712320,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":19273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2322","reported_seq":2388,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-11T04:56:18.220809+0000","last_change":"2024-09-11T04:56:18.231170+0000","last_active":"2024-09-11T04:56:18.231170+0000","last_peered":"2024-09-11T04:56:18.220809+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:55:32.933998+0000","last_became_peered":"2024-09-11T04:55:32.933998+0000","last_unstale":"2024-09-11T04:56:18.220809+0000","last_undegraded":"2024-09-11T04:55:32.919878+0000","last_fullsized":"2024-09-11T04:55:32.919516+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2341","reported_seq":2393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.971079+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:20.971079+0000","last_peered":"2024-09-11T04:56:20.971079+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:20.971079+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2341,"log_dups_size":0,"ondisk_log_size":2341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19132144,"num_objects":2336,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2244","reported_seq":2294,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.965547+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:20.965547+0000","last_peered":"2024-09-11T04:56:20.965547+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:20.965547+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18373200,"num_objects":2243,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":17943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'2079","reported_seq":2140,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:13.642870+0000","last_change":"2024-09-11T04:56:00.581036+0000","last_active":"2024-09-11T04:56:13.642870+0000","last_peered":"2024-09-11T04:56:13.642870+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:55:32.930786+0000","last_became_peered":"2024-09-11T04:55:32.930786+0000","last_unstale":"2024-09-11T04:56:13.642870+0000","last_undegraded":"2024-09-11T04:55:32.918841+0000","last_fullsized":"2024-09-11T04:55:32.918479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2079,"log_dups_size":0,"ondisk_log_size":2079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:00.876587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17021520,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":16623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2306","reported_seq":2355,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.926138+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:20.926138+0000","last_peered":"2024-09-11T04:56:20.926138+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:20.926138+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18855072,"num_objects":2302,"num_object_clones":0,"num_object_copies":9208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":18428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'2092","reported_seq":2157,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:13.580150+0000","last_change":"2024-09-11T04:56:01.507219+0000","last_active":"2024-09-11T04:56:13.580150+0000","last_peered":"2024-09-11T04:56:13.580150+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:55:32.933937+0000","last_became_peered":"2024-09-11T04:55:32.933937+0000","last_unstale":"2024-09-11T04:56:13.580150+0000","last_undegraded":"2024-09-11T04:55:32.919524+0000","last_fullsized":"2024-09-11T04:55:32.919171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2092,"log_dups_size":0,"ondisk_log_size":2092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:01.854869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17110176,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":16717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2326","reported_seq":2388,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.786312+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:20.786312+0000","last_peered":"2024-09-11T04:56:20.786312+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:20.786312+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2326,"log_dups_size":0,"ondisk_log_size":2326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19010720,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":298909888,"num_objects":36493,"num_object_clones":0,"num_object_copies":145972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36520,"num_write_kb":292042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455159808,"data_stored":455159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36547,"ondisk_log_size":36547,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296000,"kb_used_data":147688,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94075840,"statfs":{"total":96636764160,"available":96333660160,"internally_reserved":0,"allocated":151232512,"data_stored":151061702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,3,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281692,"kb_used_data":146380,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94090148,"statfs":{"total":96636764160,"available":96348311552,"internally_reserved":0,"allocated":149893120,"data_stored":149735363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":45499,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":569344,"data_stored":398534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46591698},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291764,"kb_used_data":147676,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94080076,"statfs":{"total":96636764160,"available":96337997824,"internally_reserved":0,"allocated":151220224,"data_stored":151053951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,10,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151388160,"data_stored":151388160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1150976,"data_stored":1150976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":150966272,"data_stored":150966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151654400,"data_stored":151654400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:23.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:56:23.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2315", 'reported_seq': 2356, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.885390+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:56:20.885390+0000', 'last_peered': '2024-09-11T04:56:20.885390+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:56:20.885390+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2315, 'log_dups_size': 0, 'ondisk_log_size': 2315, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18964480, 'num_objects': 2315, 'num_object_clones': 0, 'num_object_copies': 9260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2315, 'num_write_kb': 18520, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:23.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:56:23.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2293", 'reported_seq': 2353, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.848133+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:56:20.848133+0000', 'last_peered': '2024-09-11T04:56:20.848133+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:56:20.848133+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2293, 'log_dups_size': 0, 'ondisk_log_size': 2293, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18748576, 'num_objects': 2289, 'num_object_clones': 0, 'num_object_copies': 9156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2291, 'num_write_kb': 18324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 10}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:23.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "28'2224", 'reported_seq': 2288, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-11T04:56:18.211765+0000', 'last_change': '2024-09-11T04:56:18.232581+0000', 'last_active': '2024-09-11T04:56:18.232581+0000', 'last_peered': '2024-09-11T04:56:18.211765+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:55:32.933552+0000', 'last_became_peered': '2024-09-11T04:55:32.933552+0000', 'last_unstale': '2024-09-11T04:56:18.211765+0000', 'last_undegraded': '2024-09-11T04:55:32.920136+0000', 'last_fullsized': '2024-09-11T04:55:32.919825+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2224, 'log_dups_size': 0, 'ondisk_log_size': 2224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18192976, 'num_objects': 2221, 'num_object_clones': 0, 'num_object_copies': 8884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2223, 'num_write_kb': 17774, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 1, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2291", 'reported_seq': 2353, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.753538+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:56:20.753538+0000', 'last_peered': '2024-09-11T04:56:20.753538+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:56:20.753538+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2291, 'log_dups_size': 0, 'ondisk_log_size': 2291, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18732192, 'num_objects': 2287, 'num_object_clones': 0, 'num_object_copies': 9148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2289, 'num_write_kb': 18301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 9}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2263", 'reported_seq': 2316, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.881676+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:56:20.881676+0000', 'last_peered': '2024-09-11T04:56:20.881676+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:56:20.881676+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2263, 'log_dups_size': 0, 'ondisk_log_size': 2263, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18511008, 'num_objects': 2260, 'num_object_clones': 0, 'num_object_copies': 9040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2261, 'num_write_kb': 18085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 10}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2332", 'reported_seq': 2372, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.925689+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:56:20.925689+0000', 'last_peered': '2024-09-11T04:56:20.925689+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:56:20.925689+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2332, 'log_dups_size': 0, 'ondisk_log_size': 2332, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19103744, 'num_objects': 2332, 'num_object_clones': 0, 'num_object_copies': 9328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2332, 'num_write_kb': 18656, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 11}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2332", 'reported_seq': 2394, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.763669+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:56:20.763669+0000', 'last_peered': '2024-09-11T04:56:20.763669+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:56:20.763669+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2332, 'log_dups_size': 0, 'ondisk_log_size': 2332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19051680, 'num_objects': 2326, 'num_object_clones': 0, 'num_object_copies': 9304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2330, 'num_write_kb': 18620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 9}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:56:23.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2372", 'reported_seq': 2433, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.826844+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:56:20.826844+0000', 'last_peered': '2024-09-11T04:56:20.826844+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:56:20.826844+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2372, 'log_dups_size': 0, 'ondisk_log_size': 2372, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19387552, 'num_objects': 2367, 'num_object_clones': 0, 'num_object_copies': 9468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2370, 'num_write_kb': 18948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 6}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2415", 'reported_seq': 2470, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:21.025875+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:56:21.025875+0000', 'last_peered': '2024-09-11T04:56:21.025875+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:56:21.025875+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2415, 'log_dups_size': 0, 'ondisk_log_size': 2415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19712320, 'num_objects': 2407, 'num_object_clones': 0, 'num_object_copies': 9628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2412, 'num_write_kb': 19273, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 13}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "28'2322", 'reported_seq': 2388, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-11T04:56:18.220809+0000', 'last_change': '2024-09-11T04:56:18.231170+0000', 'last_active': '2024-09-11T04:56:18.231170+0000', 'last_peered': '2024-09-11T04:56:18.220809+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:55:32.933998+0000', 'last_became_peered': '2024-09-11T04:55:32.933998+0000', 'last_unstale': '2024-09-11T04:56:18.220809+0000', 'last_undegraded': '2024-09-11T04:55:32.919878+0000', 'last_fullsized': '2024-09-11T04:55:32.919516+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2322, 'log_dups_size': 0, 'ondisk_log_size': 2322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19002528, 'num_objects': 2320, 'num_object_clones': 0, 'num_object_copies': 9280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2320, 'num_write_kb': 18558, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2341", 'reported_seq': 2393, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.971079+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:56:20.971079+0000', 'last_peered': '2024-09-11T04:56:20.971079+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:56:20.971079+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2341, 'log_dups_size': 0, 'ondisk_log_size': 2341, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19132144, 'num_objects': 2336, 'num_object_clones': 0, 'num_object_copies': 9344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2336, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2338, 'num_write_kb': 18692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 13}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2244", 'reported_seq': 2294, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.965547+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:56:20.965547+0000', 'last_peered': '2024-09-11T04:56:20.965547+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:56:20.965547+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2244, 'log_dups_size': 0, 'ondisk_log_size': 2244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18373200, 'num_objects': 2243, 'num_object_clones': 0, 'num_object_copies': 8972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2243, 'num_write_kb': 17943, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 57344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 15}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:56:23.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "27'2079", 'reported_seq': 2140, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-11T04:56:13.642870+0000', 'last_change': '2024-09-11T04:56:00.581036+0000', 'last_active': '2024-09-11T04:56:13.642870+0000', 'last_peered': '2024-09-11T04:56:13.642870+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:55:32.930786+0000', 'last_became_peered': '2024-09-11T04:55:32.930786+0000', 'last_unstale': '2024-09-11T04:56:13.642870+0000', 'last_undegraded': '2024-09-11T04:55:32.918841+0000', 'last_fullsized': '2024-09-11T04:55:32.918479+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2079, 'log_dups_size': 0, 'ondisk_log_size': 2079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-11T04:57:00.876587+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17021520, 'num_objects': 2078, 'num_object_clones': 0, 'num_object_copies': 8312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2078, 'num_write_kb': 16623, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2078}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:23.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:56:23.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2306", 'reported_seq': 2355, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.926138+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:56:20.926138+0000', 'last_peered': '2024-09-11T04:56:20.926138+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:56:20.926138+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2306, 'log_dups_size': 0, 'ondisk_log_size': 2306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18855072, 'num_objects': 2302, 'num_object_clones': 0, 'num_object_copies': 9208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2304, 'num_write_kb': 18428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 15}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:23.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:56:23.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "27'2092", 'reported_seq': 2157, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-11T04:56:13.580150+0000', 'last_change': '2024-09-11T04:56:01.507219+0000', 'last_active': '2024-09-11T04:56:13.580150+0000', 'last_peered': '2024-09-11T04:56:13.580150+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:55:32.933937+0000', 'last_became_peered': '2024-09-11T04:55:32.933937+0000', 'last_unstale': '2024-09-11T04:56:13.580150+0000', 'last_undegraded': '2024-09-11T04:55:32.919524+0000', 'last_fullsized': '2024-09-11T04:55:32.919171+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2092, 'log_dups_size': 0, 'ondisk_log_size': 2092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-11T04:57:01.854869+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17110176, 'num_objects': 2089, 'num_object_clones': 0, 'num_object_copies': 8356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2089, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2090, 'num_write_kb': 16717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2089}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:23.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:56:23.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2326", 'reported_seq': 2388, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:20.786312+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:56:20.786312+0000', 'last_peered': '2024-09-11T04:56:20.786312+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:56:20.786312+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2326, 'log_dups_size': 0, 'ondisk_log_size': 2326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19010720, 'num_objects': 2321, 'num_object_clones': 0, 'num_object_copies': 9284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2324, 'num_write_kb': 18580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 10}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:23.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:56:23.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:56:23.688088+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.021018 2024-09-11T04:56:23.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:56:23.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 40 20 38047 38027 7.42625 2.94531 0.0461149 0.021018 2024-09-11T04:56:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:24.263 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31848/145972 objects degraded (21.818%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":111,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+undersized+degraded","count":2},{"state_name":"peering","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36493,"data_bytes":298909888,"bytes_used":937512960,"bytes_avail":385609543680,"bytes_total":386547056640,"inactive_pgs_ratio":0.125,"degraded_objects":31848,"degraded_total":145972,"degraded_ratio":0.21817882881648532,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":9555,"recovering_keys_per_sec":0,"num_objects_recovered":7,"num_bytes_recovered":57344,"num_keys_recovered":0,"write_bytes_sec":2581244,"read_op_per_sec":0,"write_op_per_sec":315},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-11T04:56:21.617441+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:56:24.285 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:56:24.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:24.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 41 20 38433 38413 7.31867 3.01562 0.0420919 0.0213306 2024-09-11T04:56:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:24.896 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:24.924 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-11T04:56:23.617871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298909888,"num_objects":36493,"num_object_clones":0,"num_object_copies":145972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36520,"num_write_kb":292042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36547,"ondisk_log_size":36547,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":915540,"kb_used_data":442300,"kb_used_omap":14,"kb_used_meta":473137,"kb_avail":376571820,"statfs":{"total":386547056640,"available":385609543680,"internally_reserved":0,"allocated":452915200,"data_stored":452249550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":484492854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,5,3,10,1,1,0,10,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15491072,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1891,"num_write_kb":15128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001396"},"pg_stats":[{"pgid":"1.f","version":"31'2315","reported_seq":2356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.885390+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:20.885390+0000","last_peered":"2024-09-11T04:56:20.885390+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:20.885390+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18964480,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":18520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2293","reported_seq":2353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.848133+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:20.848133+0000","last_peered":"2024-09-11T04:56:20.848133+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:20.848133+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18748576,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":18324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2224","reported_seq":2288,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-11T04:56:18.211765+0000","last_change":"2024-09-11T04:56:18.232581+0000","last_active":"2024-09-11T04:56:18.232581+0000","last_peered":"2024-09-11T04:56:18.211765+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:55:32.933552+0000","last_became_peered":"2024-09-11T04:55:32.933552+0000","last_unstale":"2024-09-11T04:56:18.211765+0000","last_undegraded":"2024-09-11T04:55:32.920136+0000","last_fullsized":"2024-09-11T04:55:32.919825+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18192976,"num_objects":2221,"num_object_clones":0,"num_object_copies":8884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2223,"num_write_kb":17774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2291","reported_seq":2353,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.753538+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:20.753538+0000","last_peered":"2024-09-11T04:56:20.753538+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:20.753538+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2291,"log_dups_size":0,"ondisk_log_size":2291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18732192,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2263","reported_seq":2316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.881676+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:20.881676+0000","last_peered":"2024-09-11T04:56:20.881676+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:20.881676+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18511008,"num_objects":2260,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":18085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2332","reported_seq":2372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.925689+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:20.925689+0000","last_peered":"2024-09-11T04:56:20.925689+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:20.925689+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2332,"num_write_kb":18656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2332","reported_seq":2394,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.763669+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:20.763669+0000","last_peered":"2024-09-11T04:56:20.763669+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:20.763669+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2332,"log_dups_size":0,"ondisk_log_size":2332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19051680,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":18620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2372","reported_seq":2433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.826844+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:20.826844+0000","last_peered":"2024-09-11T04:56:20.826844+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:20.826844+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2372,"log_dups_size":0,"ondisk_log_size":2372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19387552,"num_objects":2367,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":18948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2415","reported_seq":2470,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:21.025875+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:21.025875+0000","last_peered":"2024-09-11T04:56:21.025875+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:21.025875+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19712320,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":19273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2322","reported_seq":2388,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-11T04:56:18.220809+0000","last_change":"2024-09-11T04:56:18.231170+0000","last_active":"2024-09-11T04:56:18.231170+0000","last_peered":"2024-09-11T04:56:18.220809+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:55:32.933998+0000","last_became_peered":"2024-09-11T04:55:32.933998+0000","last_unstale":"2024-09-11T04:56:18.220809+0000","last_undegraded":"2024-09-11T04:55:32.919878+0000","last_fullsized":"2024-09-11T04:55:32.919516+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19002528,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":18558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2341","reported_seq":2393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.971079+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:20.971079+0000","last_peered":"2024-09-11T04:56:20.971079+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:20.971079+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2341,"log_dups_size":0,"ondisk_log_size":2341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19132144,"num_objects":2336,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2244","reported_seq":2294,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.965547+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:20.965547+0000","last_peered":"2024-09-11T04:56:20.965547+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:20.965547+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18373200,"num_objects":2243,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":17943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'2079","reported_seq":2140,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:13.642870+0000","last_change":"2024-09-11T04:56:00.581036+0000","last_active":"2024-09-11T04:56:13.642870+0000","last_peered":"2024-09-11T04:56:13.642870+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:55:32.930786+0000","last_became_peered":"2024-09-11T04:55:32.930786+0000","last_unstale":"2024-09-11T04:56:13.642870+0000","last_undegraded":"2024-09-11T04:55:32.918841+0000","last_fullsized":"2024-09-11T04:55:32.918479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2079,"log_dups_size":0,"ondisk_log_size":2079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:00.876587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17021520,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":16623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2306","reported_seq":2355,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.926138+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:20.926138+0000","last_peered":"2024-09-11T04:56:20.926138+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:20.926138+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18855072,"num_objects":2302,"num_object_clones":0,"num_object_copies":9208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":18428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'2092","reported_seq":2157,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:56:13.580150+0000","last_change":"2024-09-11T04:56:01.507219+0000","last_active":"2024-09-11T04:56:13.580150+0000","last_peered":"2024-09-11T04:56:13.580150+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:55:32.933937+0000","last_became_peered":"2024-09-11T04:55:32.933937+0000","last_unstale":"2024-09-11T04:56:13.580150+0000","last_undegraded":"2024-09-11T04:55:32.919524+0000","last_fullsized":"2024-09-11T04:55:32.919171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2092,"log_dups_size":0,"ondisk_log_size":2092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T04:57:01.854869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17110176,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":16717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2326","reported_seq":2388,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:20.786312+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:20.786312+0000","last_peered":"2024-09-11T04:56:20.786312+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:20.786312+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2326,"log_dups_size":0,"ondisk_log_size":2326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19010720,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":298909888,"num_objects":36493,"num_object_clones":0,"num_object_copies":145972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36520,"num_write_kb":292042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455159808,"data_stored":455159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36547,"ondisk_log_size":36547,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296000,"kb_used_data":147688,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94075840,"statfs":{"total":96636764160,"available":96333660160,"internally_reserved":0,"allocated":151232512,"data_stored":151061702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,3,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281692,"kb_used_data":146380,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94090148,"statfs":{"total":96636764160,"available":96348311552,"internally_reserved":0,"allocated":149893120,"data_stored":149735363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":45499,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":569344,"data_stored":398534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":46591698},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291764,"kb_used_data":147676,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94080076,"statfs":{"total":96636764160,"available":96337997824,"internally_reserved":0,"allocated":151220224,"data_stored":151053951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,10,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151388160,"data_stored":151388160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1150976,"data_stored":1150976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":150966272,"data_stored":150966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151654400,"data_stored":151654400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:25.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 42 20 38861 38841 7.22402 3.34375 0.03186 0.0216172 2024-09-11T04:56:26.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 43 20 39283 39263 7.13268 3.29688 0.0372869 0.0218864 2024-09-11T04:56:27.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 44 20 39713 39693 7.04692 3.35938 0.0525311 0.0221565 2024-09-11T04:56:27.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:28.533 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:28.560 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-11T04:56:27.618710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317636800,"num_objects":38779,"num_object_clones":0,"num_object_copies":155116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38806,"num_write_kb":310330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38833,"ondisk_log_size":38833,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":955416,"kb_used_data":469720,"kb_used_omap":14,"kb_used_meta":485617,"kb_avail":376531944,"statfs":{"total":386547056640,"available":385568710656,"internally_reserved":0,"allocated":480993280,"data_stored":480310040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":497272374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,13,8,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30482432,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3721,"num_write_kb":29768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.408029"},"pg_stats":[{"pgid":"1.f","version":"31'2436","reported_seq":2477,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.027418+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:26.027418+0000","last_peered":"2024-09-11T04:56:26.027418+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:26.027418+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2436,"log_dups_size":0,"ondisk_log_size":2436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19955712,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2420","reported_seq":2481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.838049+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:25.838049+0000","last_peered":"2024-09-11T04:56:25.838049+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:25.838049+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2420,"log_dups_size":0,"ondisk_log_size":2420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19788960,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2418,"num_write_kb":19340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2287","reported_seq":2361,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:23.658707+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:23.658707+0000","last_peered":"2024-09-11T04:56:23.658707+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:23.658707+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2287,"log_dups_size":0,"ondisk_log_size":2287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18709072,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2286,"num_write_kb":18278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2429","reported_seq":2491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.802102+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:25.802102+0000","last_peered":"2024-09-11T04:56:25.802102+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:25.802102+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2429,"log_dups_size":0,"ondisk_log_size":2429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19862688,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2427,"num_write_kb":19405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2392","reported_seq":2445,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.932831+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:25.932831+0000","last_peered":"2024-09-11T04:56:25.932831+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:25.932831+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2392,"log_dups_size":0,"ondisk_log_size":2392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19567776,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2390,"num_write_kb":19117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2467","reported_seq":2507,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.013540+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:26.013540+0000","last_peered":"2024-09-11T04:56:26.013540+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:26.013540+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2467,"log_dups_size":0,"ondisk_log_size":2467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20209664,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2467,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2454","reported_seq":2516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.759297+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:25.759297+0000","last_peered":"2024-09-11T04:56:25.759297+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:25.759297+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2454,"log_dups_size":0,"ondisk_log_size":2454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20051104,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2501","reported_seq":2562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.826059+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:25.826059+0000","last_peered":"2024-09-11T04:56:25.826059+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:25.826059+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20444320,"num_objects":2496,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2524","reported_seq":2579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.033924+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:26.033924+0000","last_peered":"2024-09-11T04:56:26.033924+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:26.033924+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2524,"log_dups_size":0,"ondisk_log_size":2524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20605248,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2392","reported_seq":2468,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:23.536833+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:23.536833+0000","last_peered":"2024-09-11T04:56:23.536833+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:23.536833+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2392,"log_dups_size":0,"ondisk_log_size":2392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19575968,"num_objects":2390,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2390,"num_write_kb":19118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":72}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2456","reported_seq":2510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.042511+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:26.042511+0000","last_peered":"2024-09-11T04:56:26.042511+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:26.042511+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2456,"log_dups_size":0,"ondisk_log_size":2456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074224,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":19612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2376","reported_seq":2518,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.015894+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:26.015894+0000","last_peered":"2024-09-11T04:56:26.015894+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:26.015894+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2376,"log_dups_size":0,"ondisk_log_size":2376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19454544,"num_objects":2375,"num_object_clones":0,"num_object_copies":9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2375,"num_write_kb":18999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2408","reported_seq":2481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:23.404356+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:23.404356+0000","last_peered":"2024-09-11T04:56:23.404356+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:23.404356+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2408,"log_dups_size":0,"ondisk_log_size":2408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19716688,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":19255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":75}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2444","reported_seq":2493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.968445+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:25.968445+0000","last_peered":"2024-09-11T04:56:25.968445+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:25.968445+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19985568,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":19532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2386","reported_seq":2463,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:23.656506+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:23.656506+0000","last_peered":"2024-09-11T04:56:23.656506+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:23.656506+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2386,"log_dups_size":0,"ondisk_log_size":2386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19518624,"num_objects":2383,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2384,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2461","reported_seq":2523,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.835952+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:25.835952+0000","last_peered":"2024-09-11T04:56:25.835952+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:25.835952+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2461,"log_dups_size":0,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20116640,"num_objects":2456,"num_object_clones":0,"num_object_copies":9824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":317636800,"num_objects":38779,"num_object_clones":0,"num_object_copies":155116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38806,"num_write_kb":310330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":484597760,"data_stored":484597760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38833,"ondisk_log_size":38833,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295524,"kb_used_data":155540,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94076316,"statfs":{"total":96636764160,"available":96334147584,"internally_reserved":0,"allocated":159272960,"data_stored":159102150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":290684,"kb_used_data":151212,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94081156,"statfs":{"total":96636764160,"available":96339103744,"internally_reserved":0,"allocated":154841088,"data_stored":154670278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62528,"kb_used_data":8688,"kb_used_omap":4,"kb_used_meta":53819,"kb_avail":94309312,"statfs":{"total":96636764160,"available":96572735488,"internally_reserved":0,"allocated":8896512,"data_stored":8725702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55111378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306680,"kb_used_data":154280,"kb_used_omap":4,"kb_used_meta":152379,"kb_avail":94065160,"statfs":{"total":96636764160,"available":96322723840,"internally_reserved":0,"allocated":157982720,"data_stored":157811910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156036815},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159453184,"data_stored":159453184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9510912,"data_stored":9510912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155824128,"data_stored":155824128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159809536,"data_stored":159809536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:28.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:28.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 45 20 40108 40088 6.9589 3.08594 0.0886652 0.022435 2024-09-11T04:56:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:29.175 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:29.197 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-11T04:56:27.618710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317636800,"num_objects":38779,"num_object_clones":0,"num_object_copies":155116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38806,"num_write_kb":310330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38833,"ondisk_log_size":38833,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":955416,"kb_used_data":469720,"kb_used_omap":14,"kb_used_meta":485617,"kb_avail":376531944,"statfs":{"total":386547056640,"available":385568710656,"internally_reserved":0,"allocated":480993280,"data_stored":480310040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":497272374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,13,8,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30482432,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3721,"num_write_kb":29768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.408029"},"pg_stats":[{"pgid":"1.f","version":"31'2436","reported_seq":2477,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.027418+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:26.027418+0000","last_peered":"2024-09-11T04:56:26.027418+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:26.027418+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2436,"log_dups_size":0,"ondisk_log_size":2436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19955712,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2420","reported_seq":2481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.838049+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:25.838049+0000","last_peered":"2024-09-11T04:56:25.838049+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:25.838049+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2420,"log_dups_size":0,"ondisk_log_size":2420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19788960,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2418,"num_write_kb":19340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2287","reported_seq":2361,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:23.658707+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:23.658707+0000","last_peered":"2024-09-11T04:56:23.658707+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:23.658707+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2287,"log_dups_size":0,"ondisk_log_size":2287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18709072,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2286,"num_write_kb":18278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2429","reported_seq":2491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.802102+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:25.802102+0000","last_peered":"2024-09-11T04:56:25.802102+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:25.802102+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2429,"log_dups_size":0,"ondisk_log_size":2429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19862688,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2427,"num_write_kb":19405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2392","reported_seq":2445,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.932831+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:25.932831+0000","last_peered":"2024-09-11T04:56:25.932831+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:25.932831+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2392,"log_dups_size":0,"ondisk_log_size":2392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19567776,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2390,"num_write_kb":19117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2467","reported_seq":2507,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.013540+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:26.013540+0000","last_peered":"2024-09-11T04:56:26.013540+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:26.013540+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2467,"log_dups_size":0,"ondisk_log_size":2467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20209664,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2467,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2454","reported_seq":2516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.759297+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:25.759297+0000","last_peered":"2024-09-11T04:56:25.759297+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:25.759297+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2454,"log_dups_size":0,"ondisk_log_size":2454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20051104,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2501","reported_seq":2562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.826059+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:25.826059+0000","last_peered":"2024-09-11T04:56:25.826059+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:25.826059+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20444320,"num_objects":2496,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2524","reported_seq":2579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.033924+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:26.033924+0000","last_peered":"2024-09-11T04:56:26.033924+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:26.033924+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2524,"log_dups_size":0,"ondisk_log_size":2524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20605248,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2392","reported_seq":2468,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:23.536833+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:23.536833+0000","last_peered":"2024-09-11T04:56:23.536833+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:23.536833+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2392,"log_dups_size":0,"ondisk_log_size":2392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19575968,"num_objects":2390,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2390,"num_write_kb":19118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":72}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2456","reported_seq":2510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.042511+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:26.042511+0000","last_peered":"2024-09-11T04:56:26.042511+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:26.042511+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2456,"log_dups_size":0,"ondisk_log_size":2456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074224,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":19612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2376","reported_seq":2518,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.015894+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:26.015894+0000","last_peered":"2024-09-11T04:56:26.015894+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:26.015894+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2376,"log_dups_size":0,"ondisk_log_size":2376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19454544,"num_objects":2375,"num_object_clones":0,"num_object_copies":9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2375,"num_write_kb":18999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2408","reported_seq":2481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:23.404356+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:23.404356+0000","last_peered":"2024-09-11T04:56:23.404356+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:23.404356+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2408,"log_dups_size":0,"ondisk_log_size":2408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19716688,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":19255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":75}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2444","reported_seq":2493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.968445+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:25.968445+0000","last_peered":"2024-09-11T04:56:25.968445+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:25.968445+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19985568,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":19532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2386","reported_seq":2463,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:23.656506+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:23.656506+0000","last_peered":"2024-09-11T04:56:23.656506+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:23.656506+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2386,"log_dups_size":0,"ondisk_log_size":2386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19518624,"num_objects":2383,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2384,"num_write_kb":19069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2461","reported_seq":2523,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.835952+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:25.835952+0000","last_peered":"2024-09-11T04:56:25.835952+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:25.835952+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2461,"log_dups_size":0,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20116640,"num_objects":2456,"num_object_clones":0,"num_object_copies":9824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":317636800,"num_objects":38779,"num_object_clones":0,"num_object_copies":155116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38806,"num_write_kb":310330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":484597760,"data_stored":484597760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38833,"ondisk_log_size":38833,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295524,"kb_used_data":155540,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94076316,"statfs":{"total":96636764160,"available":96334147584,"internally_reserved":0,"allocated":159272960,"data_stored":159102150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":290684,"kb_used_data":151212,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94081156,"statfs":{"total":96636764160,"available":96339103744,"internally_reserved":0,"allocated":154841088,"data_stored":154670278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62528,"kb_used_data":8688,"kb_used_omap":4,"kb_used_meta":53819,"kb_avail":94309312,"statfs":{"total":96636764160,"available":96572735488,"internally_reserved":0,"allocated":8896512,"data_stored":8725702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55111378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306680,"kb_used_data":154280,"kb_used_omap":4,"kb_used_meta":152379,"kb_avail":94065160,"statfs":{"total":96636764160,"available":96322723840,"internally_reserved":0,"allocated":157982720,"data_stored":157811910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156036815},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159453184,"data_stored":159453184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9510912,"data_stored":9510912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155824128,"data_stored":155824128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159809536,"data_stored":159809536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:29.198 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:56:29.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2436", 'reported_seq': 2477, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:26.027418+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:56:26.027418+0000', 'last_peered': '2024-09-11T04:56:26.027418+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:56:26.027418+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2436, 'log_dups_size': 0, 'ondisk_log_size': 2436, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19955712, 'num_objects': 2436, 'num_object_clones': 0, 'num_object_copies': 9744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2436, 'num_write_kb': 19488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:29.198 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:56:29.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2420", 'reported_seq': 2481, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:25.838049+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:56:25.838049+0000', 'last_peered': '2024-09-11T04:56:25.838049+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:56:25.838049+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2420, 'log_dups_size': 0, 'ondisk_log_size': 2420, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19788960, 'num_objects': 2416, 'num_object_clones': 0, 'num_object_copies': 9664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2418, 'num_write_kb': 19340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 137}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2287", 'reported_seq': 2361, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:23.658707+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:56:23.658707+0000', 'last_peered': '2024-09-11T04:56:23.658707+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:56:23.658707+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2287, 'log_dups_size': 0, 'ondisk_log_size': 2287, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18709072, 'num_objects': 2284, 'num_object_clones': 0, 'num_object_copies': 9136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2286, 'num_write_kb': 18278, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 64}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2429", 'reported_seq': 2491, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:25.802102+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:56:25.802102+0000', 'last_peered': '2024-09-11T04:56:25.802102+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:56:25.802102+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2429, 'log_dups_size': 0, 'ondisk_log_size': 2429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19862688, 'num_objects': 2425, 'num_object_clones': 0, 'num_object_copies': 9700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2427, 'num_write_kb': 19405, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2392", 'reported_seq': 2445, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:25.932831+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:56:25.932831+0000', 'last_peered': '2024-09-11T04:56:25.932831+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:56:25.932831+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2392, 'log_dups_size': 0, 'ondisk_log_size': 2392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19567776, 'num_objects': 2389, 'num_object_clones': 0, 'num_object_copies': 9556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2390, 'num_write_kb': 19117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 139}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2467", 'reported_seq': 2507, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:26.013540+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:56:26.013540+0000', 'last_peered': '2024-09-11T04:56:26.013540+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:56:26.013540+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2467, 'log_dups_size': 0, 'ondisk_log_size': 2467, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20209664, 'num_objects': 2467, 'num_object_clones': 0, 'num_object_copies': 9868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2467, 'num_write_kb': 19736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 146}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:56:29.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2454", 'reported_seq': 2516, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:25.759297+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:56:25.759297+0000', 'last_peered': '2024-09-11T04:56:25.759297+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:56:25.759297+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2454, 'log_dups_size': 0, 'ondisk_log_size': 2454, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20051104, 'num_objects': 2448, 'num_object_clones': 0, 'num_object_copies': 9792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2448, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2452, 'num_write_kb': 19596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 131}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2501", 'reported_seq': 2562, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:25.826059+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:56:25.826059+0000', 'last_peered': '2024-09-11T04:56:25.826059+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:56:25.826059+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2501, 'log_dups_size': 0, 'ondisk_log_size': 2501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20444320, 'num_objects': 2496, 'num_object_clones': 0, 'num_object_copies': 9984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2499, 'num_write_kb': 19980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2524", 'reported_seq': 2579, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:26.033924+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:56:26.033924+0000', 'last_peered': '2024-09-11T04:56:26.033924+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:56:26.033924+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2524, 'log_dups_size': 0, 'ondisk_log_size': 2524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20605248, 'num_objects': 2516, 'num_object_clones': 0, 'num_object_copies': 10064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2521, 'num_write_kb': 20145, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 122}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2392", 'reported_seq': 2468, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:23.536833+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:56:23.536833+0000', 'last_peered': '2024-09-11T04:56:23.536833+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:56:23.536833+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2392, 'log_dups_size': 0, 'ondisk_log_size': 2392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19575968, 'num_objects': 2390, 'num_object_clones': 0, 'num_object_copies': 9560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2390, 'num_write_kb': 19118, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 72}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2456", 'reported_seq': 2510, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:26.042511+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:56:26.042511+0000', 'last_peered': '2024-09-11T04:56:26.042511+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:56:26.042511+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2456, 'log_dups_size': 0, 'ondisk_log_size': 2456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20074224, 'num_objects': 2451, 'num_object_clones': 0, 'num_object_copies': 9804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2453, 'num_write_kb': 19612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 128}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2376", 'reported_seq': 2518, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:26.015894+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:56:26.015894+0000', 'last_peered': '2024-09-11T04:56:26.015894+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:56:26.015894+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2376, 'log_dups_size': 0, 'ondisk_log_size': 2376, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19454544, 'num_objects': 2375, 'num_object_clones': 0, 'num_object_copies': 9500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2375, 'num_write_kb': 18999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 54, 'num_bytes_recovered': 434176, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 193}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:29.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:56:29.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2408", 'reported_seq': 2481, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:23.404356+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:56:23.404356+0000', 'last_peered': '2024-09-11T04:56:23.404356+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:56:23.404356+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2408, 'log_dups_size': 0, 'ondisk_log_size': 2408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19716688, 'num_objects': 2407, 'num_object_clones': 0, 'num_object_copies': 9628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2407, 'num_write_kb': 19255, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 75}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:29.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:56:29.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2444", 'reported_seq': 2493, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:25.968445+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:56:25.968445+0000', 'last_peered': '2024-09-11T04:56:25.968445+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:56:25.968445+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2444, 'log_dups_size': 0, 'ondisk_log_size': 2444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19985568, 'num_objects': 2440, 'num_object_clones': 0, 'num_object_copies': 9760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2442, 'num_write_kb': 19532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 153}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:29.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:56:29.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2386", 'reported_seq': 2463, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:23.656506+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:56:23.656506+0000', 'last_peered': '2024-09-11T04:56:23.656506+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:56:23.656506+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2386, 'log_dups_size': 0, 'ondisk_log_size': 2386, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19518624, 'num_objects': 2383, 'num_object_clones': 0, 'num_object_copies': 9532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2384, 'num_write_kb': 19069, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 65}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:29.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:56:29.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2461", 'reported_seq': 2523, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:25.835952+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:56:25.835952+0000', 'last_peered': '2024-09-11T04:56:25.835952+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:56:25.835952+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2461, 'log_dups_size': 0, 'ondisk_log_size': 2461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20116640, 'num_objects': 2456, 'num_object_clones': 0, 'num_object_copies': 9824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2459, 'num_write_kb': 19660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 145}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:29.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:56:29.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 46 20 40482 40462 6.87114 2.92188 0.0565925 0.0227218 2024-09-11T04:56:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:29.957 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36829/157136 objects degraded (23.438%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":116,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39284,"data_bytes":321773760,"bytes_used":990744576,"bytes_avail":385556312064,"bytes_total":386547056640,"degraded_objects":36829,"degraded_total":157136,"degraded_ratio":0.23437659097851543,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":43407,"recovering_keys_per_sec":0,"num_objects_recovered":53,"num_bytes_recovered":434176,"num_keys_recovered":0,"write_bytes_sec":3461145,"read_op_per_sec":0,"write_op_per_sec":422},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-11T04:56:21.617441+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:56:29.989 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:56:29.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:30.588 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:30.588 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:30.620 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-11T04:56:29.619264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":321773760,"num_objects":39284,"num_object_clones":0,"num_object_copies":157136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39311,"num_write_kb":314370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39338,"ondisk_log_size":39338,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":967524,"kb_used_data":477668,"kb_used_omap":14,"kb_used_meta":489777,"kb_avail":376519836,"statfs":{"total":386547056640,"available":385556312064,"internally_reserved":0,"allocated":489132032,"data_stored":488448792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":501532214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,8,13,10,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34619392,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":33808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002293"},"pg_stats":[{"pgid":"1.f","version":"31'2436","reported_seq":2477,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.027418+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:26.027418+0000","last_peered":"2024-09-11T04:56:26.027418+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:26.027418+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2436,"log_dups_size":0,"ondisk_log_size":2436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19955712,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2420","reported_seq":2481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.838049+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:25.838049+0000","last_peered":"2024-09-11T04:56:25.838049+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:25.838049+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2420,"log_dups_size":0,"ondisk_log_size":2420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19788960,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2418,"num_write_kb":19340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2408","reported_seq":2482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.525921+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:28.525921+0000","last_peered":"2024-09-11T04:56:28.525921+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:28.525921+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2408,"log_dups_size":0,"ondisk_log_size":2408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19700304,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":19246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":185}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2429","reported_seq":2491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.802102+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:25.802102+0000","last_peered":"2024-09-11T04:56:25.802102+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:25.802102+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2429,"log_dups_size":0,"ondisk_log_size":2429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19862688,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2427,"num_write_kb":19405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2392","reported_seq":2445,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.932831+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:25.932831+0000","last_peered":"2024-09-11T04:56:25.932831+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:25.932831+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2392,"log_dups_size":0,"ondisk_log_size":2392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19567776,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2390,"num_write_kb":19117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2467","reported_seq":2507,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.013540+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:26.013540+0000","last_peered":"2024-09-11T04:56:26.013540+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:26.013540+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2467,"log_dups_size":0,"ondisk_log_size":2467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20209664,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2467,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2454","reported_seq":2516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.759297+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:25.759297+0000","last_peered":"2024-09-11T04:56:25.759297+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:25.759297+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2454,"log_dups_size":0,"ondisk_log_size":2454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20051104,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2501","reported_seq":2562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.826059+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:25.826059+0000","last_peered":"2024-09-11T04:56:25.826059+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:25.826059+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20444320,"num_objects":2496,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2524","reported_seq":2579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.033924+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:26.033924+0000","last_peered":"2024-09-11T04:56:26.033924+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:26.033924+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2524,"log_dups_size":0,"ondisk_log_size":2524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20605248,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2524","reported_seq":2600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.592804+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:28.592804+0000","last_peered":"2024-09-11T04:56:28.592804+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:28.592804+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2524,"log_dups_size":0,"ondisk_log_size":2524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":204}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2456","reported_seq":2510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.042511+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:26.042511+0000","last_peered":"2024-09-11T04:56:26.042511+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:26.042511+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2456,"log_dups_size":0,"ondisk_log_size":2456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074224,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":19612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2376","reported_seq":2518,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:26.015894+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:26.015894+0000","last_peered":"2024-09-11T04:56:26.015894+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:26.015894+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2376,"log_dups_size":0,"ondisk_log_size":2376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19454544,"num_objects":2375,"num_object_clones":0,"num_object_copies":9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2375,"num_write_kb":18999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":193}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2523","reported_seq":2595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.595316+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:28.595316+0000","last_peered":"2024-09-11T04:56:28.595316+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:28.595316+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658768,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":190}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2444","reported_seq":2493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.968445+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:25.968445+0000","last_peered":"2024-09-11T04:56:25.968445+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:25.968445+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19985568,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":19532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2523","reported_seq":2600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.634959+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:28.634959+0000","last_peered":"2024-09-11T04:56:28.634959+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:28.634959+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20640928,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2461","reported_seq":2523,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:25.835952+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:25.835952+0000","last_peered":"2024-09-11T04:56:25.835952+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:25.835952+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2461,"log_dups_size":0,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20116640,"num_objects":2456,"num_object_clones":0,"num_object_copies":9824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":321773760,"num_objects":39284,"num_object_clones":0,"num_object_copies":157136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39311,"num_write_kb":314370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":493056000,"data_stored":493056000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39338,"ondisk_log_size":39338,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295524,"kb_used_data":155540,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94076316,"statfs":{"total":96636764160,"available":96334147584,"internally_reserved":0,"allocated":159272960,"data_stored":159102150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302792,"kb_used_data":159160,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94069048,"statfs":{"total":96636764160,"available":96326705152,"internally_reserved":0,"allocated":162979840,"data_stored":162809030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,6,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62528,"kb_used_data":8688,"kb_used_omap":4,"kb_used_meta":53819,"kb_avail":94309312,"statfs":{"total":96636764160,"available":96572735488,"internally_reserved":0,"allocated":8896512,"data_stored":8725702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":55111378},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306680,"kb_used_data":154280,"kb_used_omap":4,"kb_used_meta":152379,"kb_avail":94065160,"statfs":{"total":96636764160,"available":96322723840,"internally_reserved":0,"allocated":157982720,"data_stored":157811910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156036815},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159453184,"data_stored":159453184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9510912,"data_stored":9510912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164282368,"data_stored":164282368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159809536,"data_stored":159809536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:30.689 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 47 20 40859 40839 6.78741 2.94531 0.0576847 0.0230087 2024-09-11T04:56:31.689 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 48 20 41283 41263 6.71502 3.3125 0.0394137 0.0232533 2024-09-11T04:56:32.689 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 49 20 41706 41686 6.64542 3.30469 0.0213546 0.0234958 2024-09-11T04:56:33.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:33.690 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 50 20 42122 42102 6.57734 3.25 0.0483064 0.0237329 2024-09-11T04:56:34.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:34.258 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-11T04:56:33.620882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334233792,"num_objects":40805,"num_object_clones":0,"num_object_copies":163220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40832,"num_write_kb":326538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40859,"ondisk_log_size":40859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":996080,"kb_used_data":502064,"kb_used_omap":14,"kb_used_meta":493937,"kb_avail":376491280,"statfs":{"total":386547056640,"available":385527070720,"internally_reserved":0,"allocated":514113536,"data_stored":513430296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":505792054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,2,14,8,11,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35323904,"num_objects":4312,"num_object_clones":0,"num_object_copies":17248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":34496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003539"},"pg_stats":[{"pgid":"1.f","version":"31'2566","reported_seq":2607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.072839+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:31.072839+0000","last_peered":"2024-09-11T04:56:31.072839+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:31.072839+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2566,"log_dups_size":0,"ondisk_log_size":2566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21020672,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":20528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2549","reported_seq":2610,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.833733+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:30.833733+0000","last_peered":"2024-09-11T04:56:30.833733+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:30.833733+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2549,"log_dups_size":0,"ondisk_log_size":2549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20845728,"num_objects":2545,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":20372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2408","reported_seq":2482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.525921+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:28.525921+0000","last_peered":"2024-09-11T04:56:28.525921+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:28.525921+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2408,"log_dups_size":0,"ondisk_log_size":2408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19700304,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":19246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":185}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2558","reported_seq":2619,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.807612+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:30.807612+0000","last_peered":"2024-09-11T04:56:30.807612+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:30.807612+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2558,"log_dups_size":0,"ondisk_log_size":2558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20919456,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2517","reported_seq":2570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.794067+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:30.794067+0000","last_peered":"2024-09-11T04:56:30.794067+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:30.794067+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2517,"log_dups_size":0,"ondisk_log_size":2517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20591776,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":20117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2590","reported_seq":2630,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.036803+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:31.036803+0000","last_peered":"2024-09-11T04:56:31.036803+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:31.036803+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21217280,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":20720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2583","reported_seq":2645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.735397+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:30.735397+0000","last_peered":"2024-09-11T04:56:30.735397+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:30.735397+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2583,"log_dups_size":0,"ondisk_log_size":2583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21107872,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2581,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":260}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2629","reported_seq":2690,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.768114+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:30.768114+0000","last_peered":"2024-09-11T04:56:30.768114+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:30.768114+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21492896,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":21004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2644","reported_seq":2699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.043103+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:31.043103+0000","last_peered":"2024-09-11T04:56:31.043103+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:31.043103+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2644,"log_dups_size":0,"ondisk_log_size":2644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21588288,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2641,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2524","reported_seq":2600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.592804+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:28.592804+0000","last_peered":"2024-09-11T04:56:28.592804+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:28.592804+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2524,"log_dups_size":0,"ondisk_log_size":2524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":204}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2580","reported_seq":2635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.052846+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:31.052846+0000","last_peered":"2024-09-11T04:56:31.052846+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:31.052846+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2580,"log_dups_size":0,"ondisk_log_size":2580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21090032,"num_objects":2575,"num_object_clones":0,"num_object_copies":10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":20604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2491","reported_seq":2735,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.093030+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:31.093030+0000","last_peered":"2024-09-11T04:56:31.093030+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:31.093030+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20396624,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":19919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2523","reported_seq":2595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.595316+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:28.595316+0000","last_peered":"2024-09-11T04:56:28.595316+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:28.595316+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658768,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":190}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2586","reported_seq":2635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.021689+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:31.021689+0000","last_peered":"2024-09-11T04:56:31.021689+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:31.021689+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2586,"log_dups_size":0,"ondisk_log_size":2586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148832,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":20668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":295}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2523","reported_seq":2600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.634959+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:28.634959+0000","last_peered":"2024-09-11T04:56:28.634959+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:28.634959+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20640928,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2588","reported_seq":2650,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.838914+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:30.838914+0000","last_peered":"2024-09-11T04:56:30.838914+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:30.838914+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2588,"log_dups_size":0,"ondisk_log_size":2588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157024,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":20676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":334233792,"num_objects":40805,"num_object_clones":0,"num_object_copies":163220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40832,"num_write_kb":326538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":517959680,"data_stored":517959680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40859,"ondisk_log_size":40859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307608,"kb_used_data":163464,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94064232,"statfs":{"total":96636764160,"available":96321773568,"internally_reserved":0,"allocated":167387136,"data_stored":167216326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,8,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302792,"kb_used_data":159160,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94069048,"statfs":{"total":96636764160,"available":96326705152,"internally_reserved":0,"allocated":162979840,"data_stored":162809030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,6,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79068,"kb_used_data":16908,"kb_used_omap":4,"kb_used_meta":62139,"kb_avail":94292772,"statfs":{"total":96636764160,"available":96555798528,"internally_reserved":0,"allocated":17313792,"data_stored":17142982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":63631058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306612,"kb_used_data":162532,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94065228,"statfs":{"total":96636764160,"available":96322793472,"internally_reserved":0,"allocated":166432768,"data_stored":166261958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167649280,"data_stored":167649280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17965056,"data_stored":17965056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164282368,"data_stored":164282368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":168062976,"data_stored":168062976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:34.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:34.690 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 51 20 42491 42471 6.50488 2.88281 0.0311077 0.0240043 2024-09-11T04:56:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:34.834 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:34.861 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-11T04:56:33.620882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334233792,"num_objects":40805,"num_object_clones":0,"num_object_copies":163220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40832,"num_write_kb":326538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40859,"ondisk_log_size":40859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":996080,"kb_used_data":502064,"kb_used_omap":14,"kb_used_meta":493937,"kb_avail":376491280,"statfs":{"total":386547056640,"available":385527070720,"internally_reserved":0,"allocated":514113536,"data_stored":513430296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":505792054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,2,14,8,11,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35323904,"num_objects":4312,"num_object_clones":0,"num_object_copies":17248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":34496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003539"},"pg_stats":[{"pgid":"1.f","version":"31'2566","reported_seq":2607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.072839+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:31.072839+0000","last_peered":"2024-09-11T04:56:31.072839+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:31.072839+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2566,"log_dups_size":0,"ondisk_log_size":2566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21020672,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":20528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2549","reported_seq":2610,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.833733+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:30.833733+0000","last_peered":"2024-09-11T04:56:30.833733+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:30.833733+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2549,"log_dups_size":0,"ondisk_log_size":2549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20845728,"num_objects":2545,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":20372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2408","reported_seq":2482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.525921+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:28.525921+0000","last_peered":"2024-09-11T04:56:28.525921+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:28.525921+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2408,"log_dups_size":0,"ondisk_log_size":2408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19700304,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":19246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":185}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2558","reported_seq":2619,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.807612+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:30.807612+0000","last_peered":"2024-09-11T04:56:30.807612+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:30.807612+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2558,"log_dups_size":0,"ondisk_log_size":2558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20919456,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2517","reported_seq":2570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.794067+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:30.794067+0000","last_peered":"2024-09-11T04:56:30.794067+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:30.794067+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2517,"log_dups_size":0,"ondisk_log_size":2517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20591776,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":20117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2590","reported_seq":2630,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.036803+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:31.036803+0000","last_peered":"2024-09-11T04:56:31.036803+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:31.036803+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21217280,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":20720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2583","reported_seq":2645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.735397+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:30.735397+0000","last_peered":"2024-09-11T04:56:30.735397+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:30.735397+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2583,"log_dups_size":0,"ondisk_log_size":2583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21107872,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2581,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":260}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2629","reported_seq":2690,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.768114+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:30.768114+0000","last_peered":"2024-09-11T04:56:30.768114+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:30.768114+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21492896,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":21004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2644","reported_seq":2699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.043103+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:31.043103+0000","last_peered":"2024-09-11T04:56:31.043103+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:31.043103+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2644,"log_dups_size":0,"ondisk_log_size":2644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21588288,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2641,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2524","reported_seq":2600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.592804+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:28.592804+0000","last_peered":"2024-09-11T04:56:28.592804+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:28.592804+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2524,"log_dups_size":0,"ondisk_log_size":2524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657312,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":204}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2580","reported_seq":2635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.052846+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:31.052846+0000","last_peered":"2024-09-11T04:56:31.052846+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:31.052846+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2580,"log_dups_size":0,"ondisk_log_size":2580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21090032,"num_objects":2575,"num_object_clones":0,"num_object_copies":10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":20604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2491","reported_seq":2735,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.093030+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:31.093030+0000","last_peered":"2024-09-11T04:56:31.093030+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:31.093030+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20396624,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":19919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2523","reported_seq":2595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.595316+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:28.595316+0000","last_peered":"2024-09-11T04:56:28.595316+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:28.595316+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658768,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":190}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2586","reported_seq":2635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.021689+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:31.021689+0000","last_peered":"2024-09-11T04:56:31.021689+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:31.021689+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2586,"log_dups_size":0,"ondisk_log_size":2586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148832,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":20668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":295}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2523","reported_seq":2600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:28.634959+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:28.634959+0000","last_peered":"2024-09-11T04:56:28.634959+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:28.634959+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20640928,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":20165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2588","reported_seq":2650,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.838914+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:30.838914+0000","last_peered":"2024-09-11T04:56:30.838914+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:30.838914+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2588,"log_dups_size":0,"ondisk_log_size":2588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157024,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":20676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":334233792,"num_objects":40805,"num_object_clones":0,"num_object_copies":163220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40832,"num_write_kb":326538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":517959680,"data_stored":517959680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40859,"ondisk_log_size":40859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307608,"kb_used_data":163464,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94064232,"statfs":{"total":96636764160,"available":96321773568,"internally_reserved":0,"allocated":167387136,"data_stored":167216326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,8,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302792,"kb_used_data":159160,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94069048,"statfs":{"total":96636764160,"available":96326705152,"internally_reserved":0,"allocated":162979840,"data_stored":162809030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,6,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79068,"kb_used_data":16908,"kb_used_omap":4,"kb_used_meta":62139,"kb_avail":94292772,"statfs":{"total":96636764160,"available":96555798528,"internally_reserved":0,"allocated":17313792,"data_stored":17142982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":63631058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306612,"kb_used_data":162532,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94065228,"statfs":{"total":96636764160,"available":96322793472,"internally_reserved":0,"allocated":166432768,"data_stored":166261958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167649280,"data_stored":167649280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17965056,"data_stored":17965056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164282368,"data_stored":164282368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":168062976,"data_stored":168062976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:34.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:56:34.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2566", 'reported_seq': 2607, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:31.072839+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:56:31.072839+0000', 'last_peered': '2024-09-11T04:56:31.072839+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:56:31.072839+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2566, 'log_dups_size': 0, 'ondisk_log_size': 2566, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21020672, 'num_objects': 2566, 'num_object_clones': 0, 'num_object_copies': 10264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2566, 'num_write_kb': 20528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 254}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:34.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:56:34.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2549", 'reported_seq': 2610, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:30.833733+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:56:30.833733+0000', 'last_peered': '2024-09-11T04:56:30.833733+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:56:30.833733+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2549, 'log_dups_size': 0, 'ondisk_log_size': 2549, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20845728, 'num_objects': 2545, 'num_object_clones': 0, 'num_object_copies': 10180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2547, 'num_write_kb': 20372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:34.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:56:34.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2408", 'reported_seq': 2482, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:28.525921+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:56:28.525921+0000', 'last_peered': '2024-09-11T04:56:28.525921+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:56:28.525921+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2408, 'log_dups_size': 0, 'ondisk_log_size': 2408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19700304, 'num_objects': 2405, 'num_object_clones': 0, 'num_object_copies': 9620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2407, 'num_write_kb': 19246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 185}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2558", 'reported_seq': 2619, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:30.807612+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:56:30.807612+0000', 'last_peered': '2024-09-11T04:56:30.807612+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:56:30.807612+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2558, 'log_dups_size': 0, 'ondisk_log_size': 2558, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20919456, 'num_objects': 2554, 'num_object_clones': 0, 'num_object_copies': 10216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2556, 'num_write_kb': 20437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 276}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2517", 'reported_seq': 2570, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:30.794067+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:56:30.794067+0000', 'last_peered': '2024-09-11T04:56:30.794067+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:56:30.794067+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2517, 'log_dups_size': 0, 'ondisk_log_size': 2517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20591776, 'num_objects': 2514, 'num_object_clones': 0, 'num_object_copies': 10056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2515, 'num_write_kb': 20117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 264}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2590", 'reported_seq': 2630, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:31.036803+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:56:31.036803+0000', 'last_peered': '2024-09-11T04:56:31.036803+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:56:31.036803+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2590, 'log_dups_size': 0, 'ondisk_log_size': 2590, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21217280, 'num_objects': 2590, 'num_object_clones': 0, 'num_object_copies': 10360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2590, 'num_write_kb': 20720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 269}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2583", 'reported_seq': 2645, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:30.735397+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:56:30.735397+0000', 'last_peered': '2024-09-11T04:56:30.735397+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:56:30.735397+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2583, 'log_dups_size': 0, 'ondisk_log_size': 2583, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21107872, 'num_objects': 2577, 'num_object_clones': 0, 'num_object_copies': 10308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2581, 'num_write_kb': 20628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 260}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:56:34.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2629", 'reported_seq': 2690, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:30.768114+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:56:30.768114+0000', 'last_peered': '2024-09-11T04:56:30.768114+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:56:30.768114+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2629, 'log_dups_size': 0, 'ondisk_log_size': 2629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21492896, 'num_objects': 2624, 'num_object_clones': 0, 'num_object_copies': 10496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2627, 'num_write_kb': 21004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2644", 'reported_seq': 2699, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:31.043103+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:56:31.043103+0000', 'last_peered': '2024-09-11T04:56:31.043103+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:56:31.043103+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2644, 'log_dups_size': 0, 'ondisk_log_size': 2644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21588288, 'num_objects': 2636, 'num_object_clones': 0, 'num_object_copies': 10544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2641, 'num_write_kb': 21105, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 242}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2524", 'reported_seq': 2600, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:28.592804+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:56:28.592804+0000', 'last_peered': '2024-09-11T04:56:28.592804+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:56:28.592804+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2524, 'log_dups_size': 0, 'ondisk_log_size': 2524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20657312, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 10088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2522, 'num_write_kb': 20174, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 204}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2580", 'reported_seq': 2635, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:31.052846+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:56:31.052846+0000', 'last_peered': '2024-09-11T04:56:31.052846+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:56:31.052846+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2580, 'log_dups_size': 0, 'ondisk_log_size': 2580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21090032, 'num_objects': 2575, 'num_object_clones': 0, 'num_object_copies': 10300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2577, 'num_write_kb': 20604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 252}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2491", 'reported_seq': 2735, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:31.093030+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:56:31.093030+0000', 'last_peered': '2024-09-11T04:56:31.093030+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:56:31.093030+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2491, 'log_dups_size': 0, 'ondisk_log_size': 2491, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20396624, 'num_objects': 2490, 'num_object_clones': 0, 'num_object_copies': 9960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2490, 'num_write_kb': 19919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 106, 'num_bytes_recovered': 860160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 360}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2523", 'reported_seq': 2595, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:28.595316+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:56:28.595316+0000', 'last_peered': '2024-09-11T04:56:28.595316+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:56:28.595316+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2523, 'log_dups_size': 0, 'ondisk_log_size': 2523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20658768, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 10088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2522, 'num_write_kb': 20175, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 190}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:34.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:56:34.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2586", 'reported_seq': 2635, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:31.021689+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:56:31.021689+0000', 'last_peered': '2024-09-11T04:56:31.021689+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:56:31.021689+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2586, 'log_dups_size': 0, 'ondisk_log_size': 2586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21148832, 'num_objects': 2582, 'num_object_clones': 0, 'num_object_copies': 10328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2584, 'num_write_kb': 20668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 295}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:34.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:56:34.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2523", 'reported_seq': 2600, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:28.634959+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:56:28.634959+0000', 'last_peered': '2024-09-11T04:56:28.634959+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:56:28.634959+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2523, 'log_dups_size': 0, 'ondisk_log_size': 2523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20640928, 'num_objects': 2520, 'num_object_clones': 0, 'num_object_copies': 10080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2521, 'num_write_kb': 20165, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:34.865 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:56:34.865 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2588", 'reported_seq': 2650, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:30.838914+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:56:30.838914+0000', 'last_peered': '2024-09-11T04:56:30.838914+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:56:30.838914+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2588, 'log_dups_size': 0, 'ondisk_log_size': 2588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21157024, 'num_objects': 2583, 'num_object_clones': 0, 'num_object_copies': 10332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2586, 'num_write_kb': 20676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:34.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:56:35.637 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:35.637 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36777/163220 objects degraded (22.532%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":122,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40805,"data_bytes":334233792,"bytes_used":1019985920,"bytes_avail":385527070720,"bytes_total":386547056640,"degraded_objects":36777,"degraded_total":163220,"degraded_ratio":0.22532165175836294,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":66881,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"write_bytes_sec":2942790,"read_op_per_sec":0,"write_op_per_sec":359},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-11T04:56:31.620416+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:56:35.674 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:56:35.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:35.690 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 52 20 42863 42843 6.43568 2.90625 0.0420755 0.024262 2024-09-11T04:56:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:36.335 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-11T04:56:35.621311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":338141376,"num_objects":41282,"num_object_clones":0,"num_object_copies":165128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41309,"num_write_kb":330354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41336,"ondisk_log_size":41336,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1012508,"kb_used_data":510172,"kb_used_omap":14,"kb_used_meta":502257,"kb_avail":376474852,"statfs":{"total":386547056640,"available":385510248448,"internally_reserved":0,"allocated":522416128,"data_stored":521732888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":514311734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,9,14,6,9,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39231488,"num_objects":4789,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003439"},"pg_stats":[{"pgid":"1.f","version":"31'2566","reported_seq":2607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.072839+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:31.072839+0000","last_peered":"2024-09-11T04:56:31.072839+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:31.072839+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2566,"log_dups_size":0,"ondisk_log_size":2566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21020672,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":20528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2549","reported_seq":2610,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.833733+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:30.833733+0000","last_peered":"2024-09-11T04:56:30.833733+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:30.833733+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2549,"log_dups_size":0,"ondisk_log_size":2549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20845728,"num_objects":2545,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":20372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2520","reported_seq":2592,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:33.717832+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:33.717832+0000","last_peered":"2024-09-11T04:56:33.717832+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:33.717832+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2520,"log_dups_size":0,"ondisk_log_size":2520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20617808,"num_objects":2517,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2519,"num_write_kb":20142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2558","reported_seq":2619,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.807612+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:30.807612+0000","last_peered":"2024-09-11T04:56:30.807612+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:30.807612+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2558,"log_dups_size":0,"ondisk_log_size":2558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20919456,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2517","reported_seq":2570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.794067+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:30.794067+0000","last_peered":"2024-09-11T04:56:30.794067+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:30.794067+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2517,"log_dups_size":0,"ondisk_log_size":2517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20591776,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":20117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2590","reported_seq":2630,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.036803+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:31.036803+0000","last_peered":"2024-09-11T04:56:31.036803+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:31.036803+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21217280,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":20720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2583","reported_seq":2645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.735397+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:30.735397+0000","last_peered":"2024-09-11T04:56:30.735397+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:30.735397+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2583,"log_dups_size":0,"ondisk_log_size":2583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21107872,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2581,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":260}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2629","reported_seq":2690,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.768114+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:30.768114+0000","last_peered":"2024-09-11T04:56:30.768114+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:30.768114+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21492896,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":21004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2644","reported_seq":2699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.043103+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:31.043103+0000","last_peered":"2024-09-11T04:56:31.043103+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:31.043103+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2644,"log_dups_size":0,"ondisk_log_size":2644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21588288,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2641,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2630","reported_seq":2706,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:33.662364+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:33.662364+0000","last_peered":"2024-09-11T04:56:33.662364+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:33.662364+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21525664,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2628,"num_write_kb":21022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2580","reported_seq":2635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.052846+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:31.052846+0000","last_peered":"2024-09-11T04:56:31.052846+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:31.052846+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2580,"log_dups_size":0,"ondisk_log_size":2580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21090032,"num_objects":2575,"num_object_clones":0,"num_object_copies":10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":20604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2491","reported_seq":2735,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.093030+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:31.093030+0000","last_peered":"2024-09-11T04:56:31.093030+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:31.093030+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20396624,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":19919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2649","reported_seq":2722,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:33.691583+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:33.691583+0000","last_peered":"2024-09-11T04:56:33.691583+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:33.691583+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2649,"log_dups_size":0,"ondisk_log_size":2649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21690960,"num_objects":2648,"num_object_clones":0,"num_object_copies":10592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2648,"num_write_kb":21183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":316}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2586","reported_seq":2635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:31.021689+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:31.021689+0000","last_peered":"2024-09-11T04:56:31.021689+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:31.021689+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2586,"log_dups_size":0,"ondisk_log_size":2586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148832,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":20668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":295}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2656","reported_seq":2733,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:33.693982+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:33.693982+0000","last_peered":"2024-09-11T04:56:33.693982+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:33.693982+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2656,"log_dups_size":0,"ondisk_log_size":2656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730464,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":21229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2588","reported_seq":2650,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:30.838914+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:30.838914+0000","last_peered":"2024-09-11T04:56:30.838914+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:30.838914+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2588,"log_dups_size":0,"ondisk_log_size":2588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157024,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":20676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":338141376,"num_objects":41282,"num_object_clones":0,"num_object_copies":165128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41309,"num_write_kb":330354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":526274560,"data_stored":526274560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41336,"ondisk_log_size":41336,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307608,"kb_used_data":163464,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94064232,"statfs":{"total":96636764160,"available":96321773568,"internally_reserved":0,"allocated":167387136,"data_stored":167216326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,8,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319220,"kb_used_data":167268,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":94052620,"statfs":{"total":96636764160,"available":96309882880,"internally_reserved":0,"allocated":171282432,"data_stored":171111622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":155580869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79068,"kb_used_data":16908,"kb_used_omap":4,"kb_used_meta":62139,"kb_avail":94292772,"statfs":{"total":96636764160,"available":96555798528,"internally_reserved":0,"allocated":17313792,"data_stored":17142982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":63631058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306612,"kb_used_data":162532,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94065228,"statfs":{"total":96636764160,"available":96322793472,"internally_reserved":0,"allocated":166432768,"data_stored":166261958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167649280,"data_stored":167649280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17965056,"data_stored":17965056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172597248,"data_stored":172597248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":168062976,"data_stored":168062976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:36.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 53 20 43245 43225 6.37056 2.98438 0.0915119 0.0245136 2024-09-11T04:56:37.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 54 20 43659 43639 6.31248 3.23438 0.0595207 0.0247385 2024-09-11T04:56:38.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 55 20 44059 44039 6.25453 3.125 0.0251859 0.0249652 2024-09-11T04:56:39.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:39.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 56 20 44461 44441 6.19892 3.14062 0.0364597 0.02519 2024-09-11T04:56:39.973 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:39.973 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:40.003 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-11T04:56:39.622281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":354582720,"num_objects":43289,"num_object_clones":0,"num_object_copies":173156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43316,"num_write_kb":346410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43343,"ondisk_log_size":43343,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1030976,"kb_used_data":545280,"kb_used_omap":14,"kb_used_meta":485617,"kb_avail":376456384,"statfs":{"total":386547056640,"available":385491337216,"internally_reserved":0,"allocated":558366720,"data_stored":557683480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":497272374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,1,13,15,22,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36945920,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":36080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003571"},"pg_stats":[{"pgid":"1.f","version":"31'2678","reported_seq":2719,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.088939+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:36.088939+0000","last_peered":"2024-09-11T04:56:36.088939+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:36.088939+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21938176,"num_objects":2678,"num_object_clones":0,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2678,"num_write_kb":21424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":366}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2671","reported_seq":2732,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.799659+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:35.799659+0000","last_peered":"2024-09-11T04:56:35.799659+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:35.799659+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21845152,"num_objects":2667,"num_object_clones":0,"num_object_copies":10668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2647","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.706906+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:38.706906+0000","last_peered":"2024-09-11T04:56:38.706906+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:38.706906+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2647,"log_dups_size":0,"ondisk_log_size":2647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21658192,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2646,"num_write_kb":21158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2671","reported_seq":2733,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.832092+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:35.832092+0000","last_peered":"2024-09-11T04:56:35.832092+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:35.832092+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21845152,"num_objects":2667,"num_object_clones":0,"num_object_copies":10668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2664","reported_seq":2717,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.128084+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:36.128084+0000","last_peered":"2024-09-11T04:56:36.128084+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:36.128084+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2664,"log_dups_size":0,"ondisk_log_size":2664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796000,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":21293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2718","reported_seq":2758,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.019397+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:36.019397+0000","last_peered":"2024-09-11T04:56:36.019397+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:36.019397+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2718,"log_dups_size":0,"ondisk_log_size":2718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22265856,"num_objects":2718,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":21744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2712","reported_seq":2772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.786524+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:35.786524+0000","last_peered":"2024-09-11T04:56:35.786524+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:35.786524+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2712,"log_dups_size":0,"ondisk_log_size":2712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22164640,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2710,"num_write_kb":21660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2760","reported_seq":2821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.791139+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:35.791139+0000","last_peered":"2024-09-11T04:56:35.791139+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:35.791139+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2760,"log_dups_size":0,"ondisk_log_size":2760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22566048,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2758,"num_write_kb":22052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":394}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2782","reported_seq":2837,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.092277+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:36.092277+0000","last_peered":"2024-09-11T04:56:36.092277+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:36.092277+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2782,"log_dups_size":0,"ondisk_log_size":2782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22718784,"num_objects":2774,"num_object_clones":0,"num_object_copies":11096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":22209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2751","reported_seq":2827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.659731+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:38.659731+0000","last_peered":"2024-09-11T04:56:38.659731+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:38.659731+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22516896,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2692","reported_seq":2747,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.127392+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:36.127392+0000","last_peered":"2024-09-11T04:56:36.127392+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:36.127392+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2692,"log_dups_size":0,"ondisk_log_size":2692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22007536,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":364}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2622","reported_seq":2968,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.080421+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:36.080421+0000","last_peered":"2024-09-11T04:56:36.080421+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:36.080421+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21469776,"num_objects":2621,"num_object_clones":0,"num_object_copies":10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":20967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":541}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2787","reported_seq":2860,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.659736+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:38.659736+0000","last_peered":"2024-09-11T04:56:38.659736+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:38.659736+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2787,"log_dups_size":0,"ondisk_log_size":2787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22821456,"num_objects":2786,"num_object_clones":0,"num_object_copies":11144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":454}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2705","reported_seq":2753,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.079308+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:36.079308+0000","last_peered":"2024-09-11T04:56:36.079308+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:36.079308+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2705,"log_dups_size":0,"ondisk_log_size":2705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22123680,"num_objects":2701,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":21620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2772","reported_seq":2849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.580812+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:38.580812+0000","last_peered":"2024-09-11T04:56:38.580812+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:38.580812+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2772,"log_dups_size":0,"ondisk_log_size":2772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22680736,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2711","reported_seq":2773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.796028+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:35.796028+0000","last_peered":"2024-09-11T04:56:35.796028+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:35.796028+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2711,"log_dups_size":0,"ondisk_log_size":2711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22164640,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2709,"num_write_kb":21660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":354582720,"num_objects":43289,"num_object_clones":0,"num_object_copies":173156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43316,"num_write_kb":346410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":558858240,"data_stored":558858240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43343,"ondisk_log_size":43343,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319872,"kb_used_data":171568,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94051968,"statfs":{"total":96636764160,"available":96309215232,"internally_reserved":0,"allocated":175685632,"data_stored":175514822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,6,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316004,"kb_used_data":176532,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94055836,"statfs":{"total":96636764160,"available":96313176064,"internally_reserved":0,"allocated":180768768,"data_stored":180597958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74936,"kb_used_data":25256,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94296904,"statfs":{"total":96636764160,"available":96560029696,"internally_reserved":0,"allocated":25862144,"data_stored":25691334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50851538},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320164,"kb_used_data":171924,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94051676,"statfs":{"total":96636764160,"available":96308916224,"internally_reserved":0,"allocated":176050176,"data_stored":175879366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":175808512,"data_stored":175808512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26312704,"data_stored":26312704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180518912,"data_stored":180518912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176218112,"data_stored":176218112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:40.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:40.653 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:40.672 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-11T04:56:39.622281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":354582720,"num_objects":43289,"num_object_clones":0,"num_object_copies":173156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43316,"num_write_kb":346410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43343,"ondisk_log_size":43343,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1030976,"kb_used_data":545280,"kb_used_omap":14,"kb_used_meta":485617,"kb_avail":376456384,"statfs":{"total":386547056640,"available":385491337216,"internally_reserved":0,"allocated":558366720,"data_stored":557683480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":497272374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,1,13,15,22,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36945920,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":36080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003571"},"pg_stats":[{"pgid":"1.f","version":"31'2678","reported_seq":2719,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.088939+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:36.088939+0000","last_peered":"2024-09-11T04:56:36.088939+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:36.088939+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21938176,"num_objects":2678,"num_object_clones":0,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2678,"num_write_kb":21424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":366}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2671","reported_seq":2732,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.799659+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:35.799659+0000","last_peered":"2024-09-11T04:56:35.799659+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:35.799659+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21845152,"num_objects":2667,"num_object_clones":0,"num_object_copies":10668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2647","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.706906+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:38.706906+0000","last_peered":"2024-09-11T04:56:38.706906+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:38.706906+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2647,"log_dups_size":0,"ondisk_log_size":2647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21658192,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2646,"num_write_kb":21158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2671","reported_seq":2733,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.832092+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:35.832092+0000","last_peered":"2024-09-11T04:56:35.832092+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:35.832092+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21845152,"num_objects":2667,"num_object_clones":0,"num_object_copies":10668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":21341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2664","reported_seq":2717,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.128084+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:36.128084+0000","last_peered":"2024-09-11T04:56:36.128084+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:36.128084+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2664,"log_dups_size":0,"ondisk_log_size":2664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796000,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":21293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2718","reported_seq":2758,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.019397+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:36.019397+0000","last_peered":"2024-09-11T04:56:36.019397+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:36.019397+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2718,"log_dups_size":0,"ondisk_log_size":2718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22265856,"num_objects":2718,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":21744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2712","reported_seq":2772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.786524+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:35.786524+0000","last_peered":"2024-09-11T04:56:35.786524+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:35.786524+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2712,"log_dups_size":0,"ondisk_log_size":2712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22164640,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2710,"num_write_kb":21660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2760","reported_seq":2821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.791139+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:35.791139+0000","last_peered":"2024-09-11T04:56:35.791139+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:35.791139+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2760,"log_dups_size":0,"ondisk_log_size":2760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22566048,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2758,"num_write_kb":22052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":394}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2782","reported_seq":2837,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.092277+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:36.092277+0000","last_peered":"2024-09-11T04:56:36.092277+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:36.092277+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2782,"log_dups_size":0,"ondisk_log_size":2782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22718784,"num_objects":2774,"num_object_clones":0,"num_object_copies":11096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":22209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2751","reported_seq":2827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.659731+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:38.659731+0000","last_peered":"2024-09-11T04:56:38.659731+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:38.659731+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22516896,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2692","reported_seq":2747,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.127392+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:36.127392+0000","last_peered":"2024-09-11T04:56:36.127392+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:36.127392+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2692,"log_dups_size":0,"ondisk_log_size":2692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22007536,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":364}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2622","reported_seq":2968,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.080421+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:36.080421+0000","last_peered":"2024-09-11T04:56:36.080421+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:36.080421+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21469776,"num_objects":2621,"num_object_clones":0,"num_object_copies":10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":20967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":541}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2787","reported_seq":2860,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.659736+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:38.659736+0000","last_peered":"2024-09-11T04:56:38.659736+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:38.659736+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2787,"log_dups_size":0,"ondisk_log_size":2787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22821456,"num_objects":2786,"num_object_clones":0,"num_object_copies":11144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":454}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2705","reported_seq":2753,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:36.079308+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:36.079308+0000","last_peered":"2024-09-11T04:56:36.079308+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:36.079308+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2705,"log_dups_size":0,"ondisk_log_size":2705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22123680,"num_objects":2701,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":21620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2772","reported_seq":2849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.580812+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:38.580812+0000","last_peered":"2024-09-11T04:56:38.580812+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:38.580812+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2772,"log_dups_size":0,"ondisk_log_size":2772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22680736,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2711","reported_seq":2773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:35.796028+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:35.796028+0000","last_peered":"2024-09-11T04:56:35.796028+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:35.796028+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2711,"log_dups_size":0,"ondisk_log_size":2711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22164640,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2709,"num_write_kb":21660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":354582720,"num_objects":43289,"num_object_clones":0,"num_object_copies":173156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43316,"num_write_kb":346410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":558858240,"data_stored":558858240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43343,"ondisk_log_size":43343,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319872,"kb_used_data":171568,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94051968,"statfs":{"total":96636764160,"available":96309215232,"internally_reserved":0,"allocated":175685632,"data_stored":175514822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,6,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316004,"kb_used_data":176532,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94055836,"statfs":{"total":96636764160,"available":96313176064,"internally_reserved":0,"allocated":180768768,"data_stored":180597958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74936,"kb_used_data":25256,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94296904,"statfs":{"total":96636764160,"available":96560029696,"internally_reserved":0,"allocated":25862144,"data_stored":25691334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":50851538},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320164,"kb_used_data":171924,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94051676,"statfs":{"total":96636764160,"available":96308916224,"internally_reserved":0,"allocated":176050176,"data_stored":175879366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":175808512,"data_stored":175808512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26312704,"data_stored":26312704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180518912,"data_stored":180518912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176218112,"data_stored":176218112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:40.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:56:40.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2678", 'reported_seq': 2719, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:36.088939+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:56:36.088939+0000', 'last_peered': '2024-09-11T04:56:36.088939+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:56:36.088939+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2678, 'log_dups_size': 0, 'ondisk_log_size': 2678, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21938176, 'num_objects': 2678, 'num_object_clones': 0, 'num_object_copies': 10712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2678, 'num_write_kb': 21424, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 366}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:40.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:56:40.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2671", 'reported_seq': 2732, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:35.799659+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:56:35.799659+0000', 'last_peered': '2024-09-11T04:56:35.799659+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:56:35.799659+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2671, 'log_dups_size': 0, 'ondisk_log_size': 2671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21845152, 'num_objects': 2667, 'num_object_clones': 0, 'num_object_copies': 10668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2669, 'num_write_kb': 21348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 388}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:40.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:56:40.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2647", 'reported_seq': 2721, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:38.706906+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:56:38.706906+0000', 'last_peered': '2024-09-11T04:56:38.706906+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:56:38.706906+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2647, 'log_dups_size': 0, 'ondisk_log_size': 2647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21658192, 'num_objects': 2644, 'num_object_clones': 0, 'num_object_copies': 10576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2646, 'num_write_kb': 21158, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 424}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2671", 'reported_seq': 2733, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:35.832092+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:56:35.832092+0000', 'last_peered': '2024-09-11T04:56:35.832092+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:56:35.832092+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2671, 'log_dups_size': 0, 'ondisk_log_size': 2671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21845152, 'num_objects': 2667, 'num_object_clones': 0, 'num_object_copies': 10668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2669, 'num_write_kb': 21341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2664", 'reported_seq': 2717, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:36.128084+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:56:36.128084+0000', 'last_peered': '2024-09-11T04:56:36.128084+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:56:36.128084+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2664, 'log_dups_size': 0, 'ondisk_log_size': 2664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21796000, 'num_objects': 2661, 'num_object_clones': 0, 'num_object_copies': 10644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2662, 'num_write_kb': 21293, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 411}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2718", 'reported_seq': 2758, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:36.019397+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:56:36.019397+0000', 'last_peered': '2024-09-11T04:56:36.019397+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:56:36.019397+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2718, 'log_dups_size': 0, 'ondisk_log_size': 2718, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22265856, 'num_objects': 2718, 'num_object_clones': 0, 'num_object_copies': 10872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2718, 'num_write_kb': 21744, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 397}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2712", 'reported_seq': 2772, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:35.786524+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:56:35.786524+0000', 'last_peered': '2024-09-11T04:56:35.786524+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:56:35.786524+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2712, 'log_dups_size': 0, 'ondisk_log_size': 2712, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22164640, 'num_objects': 2706, 'num_object_clones': 0, 'num_object_copies': 10824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2710, 'num_write_kb': 21660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 389}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:56:40.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2760", 'reported_seq': 2821, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:35.791139+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:56:35.791139+0000', 'last_peered': '2024-09-11T04:56:35.791139+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:56:35.791139+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2760, 'log_dups_size': 0, 'ondisk_log_size': 2760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22566048, 'num_objects': 2755, 'num_object_clones': 0, 'num_object_copies': 11020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2758, 'num_write_kb': 22052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 394}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:40.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:56:40.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2782", 'reported_seq': 2837, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:36.092277+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:56:36.092277+0000', 'last_peered': '2024-09-11T04:56:36.092277+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:56:36.092277+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2782, 'log_dups_size': 0, 'ondisk_log_size': 2782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22718784, 'num_objects': 2774, 'num_object_clones': 0, 'num_object_copies': 11096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2779, 'num_write_kb': 22209, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 380}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:40.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:56:40.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2751", 'reported_seq': 2827, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:38.659731+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:56:38.659731+0000', 'last_peered': '2024-09-11T04:56:38.659731+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:56:38.659731+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2751, 'log_dups_size': 0, 'ondisk_log_size': 2751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22516896, 'num_objects': 2749, 'num_object_clones': 0, 'num_object_copies': 10996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2749, 'num_write_kb': 21990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 431}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:40.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:56:40.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2692", 'reported_seq': 2747, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:36.127392+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:56:36.127392+0000', 'last_peered': '2024-09-11T04:56:36.127392+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:56:36.127392+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2692, 'log_dups_size': 0, 'ondisk_log_size': 2692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22007536, 'num_objects': 2687, 'num_object_clones': 0, 'num_object_copies': 10748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2689, 'num_write_kb': 21500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 364}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:40.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:56:40.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2622", 'reported_seq': 2968, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:36.080421+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:56:36.080421+0000', 'last_peered': '2024-09-11T04:56:36.080421+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:56:36.080421+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2622, 'log_dups_size': 0, 'ondisk_log_size': 2622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21469776, 'num_objects': 2621, 'num_object_clones': 0, 'num_object_copies': 10484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2080, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2621, 'num_write_kb': 20967, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 156, 'num_bytes_recovered': 1269760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 541}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:40.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:56:40.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2787", 'reported_seq': 2860, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:38.659736+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:56:38.659736+0000', 'last_peered': '2024-09-11T04:56:38.659736+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:56:38.659736+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2787, 'log_dups_size': 0, 'ondisk_log_size': 2787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22821456, 'num_objects': 2786, 'num_object_clones': 0, 'num_object_copies': 11144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2786, 'num_write_kb': 22287, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 454}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:40.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:56:40.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2705", 'reported_seq': 2753, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:36.079308+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:56:36.079308+0000', 'last_peered': '2024-09-11T04:56:36.079308+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:56:36.079308+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2705, 'log_dups_size': 0, 'ondisk_log_size': 2705, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22123680, 'num_objects': 2701, 'num_object_clones': 0, 'num_object_copies': 10804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2703, 'num_write_kb': 21620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:40.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:56:40.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2772", 'reported_seq': 2849, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:38.580812+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:56:38.580812+0000', 'last_peered': '2024-09-11T04:56:38.580812+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:56:38.580812+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2772, 'log_dups_size': 0, 'ondisk_log_size': 2772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22680736, 'num_objects': 2769, 'num_object_clones': 0, 'num_object_copies': 11076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2770, 'num_write_kb': 22157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:40.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:56:40.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2711", 'reported_seq': 2773, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:35.796028+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:56:35.796028+0000', 'last_peered': '2024-09-11T04:56:35.796028+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:56:35.796028+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2711, 'log_dups_size': 0, 'ondisk_log_size': 2711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22164640, 'num_objects': 2706, 'num_object_clones': 0, 'num_object_copies': 10824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2709, 'num_write_kb': 21660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:40.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:56:40.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 57 20 44823 44803 6.13978 2.82812 0.0387996 0.0254241 2024-09-11T04:56:41.448 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:41.449 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36727/173156 objects degraded (21.210%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":128,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43289,"data_bytes":354582720,"bytes_used":1055719424,"bytes_avail":385491337216,"bytes_total":386547056640,"degraded_objects":36727,"degraded_total":173156,"degraded_ratio":0.21210353669523435,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":69611,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":835584,"num_keys_recovered":0,"write_bytes_sec":3077910,"read_op_per_sec":0,"write_op_per_sec":375},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-11T04:56:31.620416+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:56:41.476 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:56:41.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:41.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 58 20 45190 45170 6.08336 2.86719 0.081083 0.0256709 2024-09-11T04:56:42.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:42.157 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:42.186 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-11T04:56:41.625714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":366616768,"num_objects":44758,"num_object_clones":0,"num_object_copies":179032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44785,"num_write_kb":358162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44812,"ondisk_log_size":44812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1060352,"kb_used_data":569088,"kb_used_omap":14,"kb_used_meta":491185,"kb_avail":376427008,"statfs":{"total":386547056640,"available":385461256192,"internally_reserved":0,"allocated":582746112,"data_stored":582062872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":502974006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,7,11,9,14,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44843008,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006449"},"pg_stats":[{"pgid":"1.f","version":"31'2802","reported_seq":2843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.233052+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:41.233052+0000","last_peered":"2024-09-11T04:56:41.233052+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:41.233052+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22953984,"num_objects":2802,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2802,"num_write_kb":22416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2797","reported_seq":2857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.777787+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:40.777787+0000","last_peered":"2024-09-11T04:56:40.777787+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:40.777787+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2797,"log_dups_size":0,"ondisk_log_size":2797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22877344,"num_objects":2793,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2795,"num_write_kb":22356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":514}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2647","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.706906+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:38.706906+0000","last_peered":"2024-09-11T04:56:38.706906+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:38.706906+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2647,"log_dups_size":0,"ondisk_log_size":2647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21658192,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2646,"num_write_kb":21158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2793","reported_seq":2855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.880240+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:40.880240+0000","last_peered":"2024-09-11T04:56:40.880240+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:40.880240+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22844576,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2791,"num_write_kb":22317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2778","reported_seq":2831,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.174553+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:41.174553+0000","last_peered":"2024-09-11T04:56:41.174553+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:41.174553+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2778,"log_dups_size":0,"ondisk_log_size":2778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22729888,"num_objects":2775,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2776,"num_write_kb":22205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2833","reported_seq":2872,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.240296+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:41.240296+0000","last_peered":"2024-09-11T04:56:41.240296+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:41.240296+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2833,"log_dups_size":0,"ondisk_log_size":2833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23207936,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":22664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2850","reported_seq":2912,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.843089+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:40.843089+0000","last_peered":"2024-09-11T04:56:40.843089+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:40.843089+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23295136,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":527}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2867","reported_seq":2928,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.827166+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:40.827166+0000","last_peered":"2024-09-11T04:56:40.827166+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:40.827166+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23442592,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2899","reported_seq":2954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.204431+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:41.204431+0000","last_peered":"2024-09-11T04:56:41.204431+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:41.204431+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23677248,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":23145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2751","reported_seq":2827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.659731+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:38.659731+0000","last_peered":"2024-09-11T04:56:38.659731+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:38.659731+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22516896,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2827","reported_seq":2881,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.162878+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:41.162878+0000","last_peered":"2024-09-11T04:56:41.162878+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:41.162878+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23113456,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":22580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2743","reported_seq":3199,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.219379+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:41.219379+0000","last_peered":"2024-09-11T04:56:41.219379+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:41.219379+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2743,"log_dups_size":0,"ondisk_log_size":2743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22461008,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":21935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2787","reported_seq":2860,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.659736+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:38.659736+0000","last_peered":"2024-09-11T04:56:38.659736+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:38.659736+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2787,"log_dups_size":0,"ondisk_log_size":2787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22821456,"num_objects":2786,"num_object_clones":0,"num_object_copies":11144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":454}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2830","reported_seq":2879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.187692+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:41.187692+0000","last_peered":"2024-09-11T04:56:41.187692+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:41.187692+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2830,"log_dups_size":0,"ondisk_log_size":2830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23147680,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2828,"num_write_kb":22620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":539}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2772","reported_seq":2849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:38.580812+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:38.580812+0000","last_peered":"2024-09-11T04:56:38.580812+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:38.580812+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2772,"log_dups_size":0,"ondisk_log_size":2772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22680736,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":451}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2836","reported_seq":2898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.709361+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:40.709361+0000","last_peered":"2024-09-11T04:56:40.709361+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:40.709361+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2836,"log_dups_size":0,"ondisk_log_size":2836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188640,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":22660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":366616768,"num_objects":44758,"num_object_clones":0,"num_object_copies":179032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44785,"num_write_kb":358162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":583548928,"data_stored":583548928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44812,"ondisk_log_size":44812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315280,"kb_used_data":179456,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94056560,"statfs":{"total":96636764160,"available":96313917440,"internally_reserved":0,"allocated":183762944,"data_stored":183592134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,4,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316004,"kb_used_data":176532,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94055836,"statfs":{"total":96636764160,"available":96313176064,"internally_reserved":0,"allocated":180768768,"data_stored":180597958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109348,"kb_used_data":33300,"kb_used_omap":4,"kb_used_meta":76027,"kb_avail":94262492,"statfs":{"total":96636764160,"available":96524791808,"internally_reserved":0,"allocated":34099200,"data_stored":33928390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":77852370},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319720,"kb_used_data":179800,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94052120,"statfs":{"total":96636764160,"available":96309370880,"internally_reserved":0,"allocated":184115200,"data_stored":183944390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":183877632,"data_stored":183877632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34717696,"data_stored":34717696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180518912,"data_stored":180518912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184434688,"data_stored":184434688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:42.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 59 20 45567 45547 6.03017 2.94531 0.0607163 0.0258985 2024-09-11T04:56:43.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:56:43.692569+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0261006 2024-09-11T04:56:43.692 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:56:43.692 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 60 20 45974 45954 5.98266 3.17969 0.032459 0.0261006 2024-09-11T04:56:44.693 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 61 20 46377 46357 5.93613 3.14844 0.0716106 0.0263061 2024-09-11T04:56:45.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:45.692 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 62 20 46750 46730 5.88738 2.91406 0.0505765 0.0265229 2024-09-11T04:56:45.764 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:45.765 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:45.788 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-11T04:56:45.626619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":370458816,"num_objects":45227,"num_object_clones":0,"num_object_copies":180908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45254,"num_write_kb":361914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45281,"ondisk_log_size":45281,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1072060,"kb_used_data":576636,"kb_used_omap":14,"kb_used_meta":495345,"kb_avail":376415300,"statfs":{"total":386547056640,"available":385449267200,"internally_reserved":0,"allocated":590475264,"data_stored":589792024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":507233846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,13,9,11,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36225024,"num_objects":4422,"num_object_clones":0,"num_object_copies":17688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":35376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005736"},"pg_stats":[{"pgid":"1.f","version":"31'2802","reported_seq":2843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.233052+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:41.233052+0000","last_peered":"2024-09-11T04:56:41.233052+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:41.233052+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22953984,"num_objects":2802,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2802,"num_write_kb":22416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2797","reported_seq":2857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.777787+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:40.777787+0000","last_peered":"2024-09-11T04:56:40.777787+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:40.777787+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2797,"log_dups_size":0,"ondisk_log_size":2797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22877344,"num_objects":2793,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2795,"num_write_kb":22356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":514}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2771","reported_seq":2845,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.691069+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:43.691069+0000","last_peered":"2024-09-11T04:56:43.691069+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:43.691069+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22674000,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2793","reported_seq":2855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.880240+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:40.880240+0000","last_peered":"2024-09-11T04:56:40.880240+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:40.880240+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22844576,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2791,"num_write_kb":22317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2778","reported_seq":2831,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.174553+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:41.174553+0000","last_peered":"2024-09-11T04:56:41.174553+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:41.174553+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2778,"log_dups_size":0,"ondisk_log_size":2778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22729888,"num_objects":2775,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2776,"num_write_kb":22205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2833","reported_seq":2872,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.240296+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:41.240296+0000","last_peered":"2024-09-11T04:56:41.240296+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:41.240296+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2833,"log_dups_size":0,"ondisk_log_size":2833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23207936,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":22664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2850","reported_seq":2912,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.843089+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:40.843089+0000","last_peered":"2024-09-11T04:56:40.843089+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:40.843089+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23295136,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":527}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2867","reported_seq":2928,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.827166+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:40.827166+0000","last_peered":"2024-09-11T04:56:40.827166+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:40.827166+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23442592,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2899","reported_seq":2954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.204431+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:41.204431+0000","last_peered":"2024-09-11T04:56:41.204431+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:41.204431+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23677248,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":23145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2854","reported_seq":2930,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.688384+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:43.688384+0000","last_peered":"2024-09-11T04:56:43.688384+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:43.688384+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23360672,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":22814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2827","reported_seq":2881,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.162878+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:41.162878+0000","last_peered":"2024-09-11T04:56:41.162878+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:41.162878+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23113456,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":22580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2743","reported_seq":3199,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.219379+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:41.219379+0000","last_peered":"2024-09-11T04:56:41.219379+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:41.219379+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2743,"log_dups_size":0,"ondisk_log_size":2743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22461008,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":21935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2908","reported_seq":2981,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.651640+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:43.651640+0000","last_peered":"2024-09-11T04:56:43.651640+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:43.651640+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2908,"log_dups_size":0,"ondisk_log_size":2908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23812688,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2907,"num_write_kb":23255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2830","reported_seq":2879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.187692+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:41.187692+0000","last_peered":"2024-09-11T04:56:41.187692+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:41.187692+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2830,"log_dups_size":0,"ondisk_log_size":2830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23147680,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2828,"num_write_kb":22620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":539}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2893","reported_seq":2969,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.694866+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:43.694866+0000","last_peered":"2024-09-11T04:56:43.694866+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:43.694866+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2893,"log_dups_size":0,"ondisk_log_size":2893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23671968,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2891,"num_write_kb":23125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":572}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2836","reported_seq":2898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.709361+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:40.709361+0000","last_peered":"2024-09-11T04:56:40.709361+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:40.709361+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2836,"log_dups_size":0,"ondisk_log_size":2836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188640,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":22660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":370458816,"num_objects":45227,"num_object_clones":0,"num_object_copies":180908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45254,"num_write_kb":361914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":591400960,"data_stored":591400960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45281,"ondisk_log_size":45281,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315280,"kb_used_data":179456,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94056560,"statfs":{"total":96636764160,"available":96313917440,"internally_reserved":0,"allocated":183762944,"data_stored":183592134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,4,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327712,"kb_used_data":184080,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94044128,"statfs":{"total":96636764160,"available":96301187072,"internally_reserved":0,"allocated":188497920,"data_stored":188327110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109348,"kb_used_data":33300,"kb_used_omap":4,"kb_used_meta":76027,"kb_avail":94262492,"statfs":{"total":96636764160,"available":96524791808,"internally_reserved":0,"allocated":34099200,"data_stored":33928390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":77852370},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319720,"kb_used_data":179800,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94052120,"statfs":{"total":96636764160,"available":96309370880,"internally_reserved":0,"allocated":184115200,"data_stored":183944390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":183877632,"data_stored":183877632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34717696,"data_stored":34717696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":188370944,"data_stored":188370944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184434688,"data_stored":184434688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:45.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:46.427 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:46.452 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-11T04:56:45.626619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":370458816,"num_objects":45227,"num_object_clones":0,"num_object_copies":180908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45254,"num_write_kb":361914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45281,"ondisk_log_size":45281,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1072060,"kb_used_data":576636,"kb_used_omap":14,"kb_used_meta":495345,"kb_avail":376415300,"statfs":{"total":386547056640,"available":385449267200,"internally_reserved":0,"allocated":590475264,"data_stored":589792024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":507233846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,13,9,11,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36225024,"num_objects":4422,"num_object_clones":0,"num_object_copies":17688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":35376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005736"},"pg_stats":[{"pgid":"1.f","version":"31'2802","reported_seq":2843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.233052+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:41.233052+0000","last_peered":"2024-09-11T04:56:41.233052+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:41.233052+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22953984,"num_objects":2802,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2802,"num_write_kb":22416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2797","reported_seq":2857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.777787+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:40.777787+0000","last_peered":"2024-09-11T04:56:40.777787+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:40.777787+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2797,"log_dups_size":0,"ondisk_log_size":2797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22877344,"num_objects":2793,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2795,"num_write_kb":22356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":514}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2771","reported_seq":2845,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.691069+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:43.691069+0000","last_peered":"2024-09-11T04:56:43.691069+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:43.691069+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22674000,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2793","reported_seq":2855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.880240+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:40.880240+0000","last_peered":"2024-09-11T04:56:40.880240+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:40.880240+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22844576,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2791,"num_write_kb":22317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2778","reported_seq":2831,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.174553+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:41.174553+0000","last_peered":"2024-09-11T04:56:41.174553+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:41.174553+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2778,"log_dups_size":0,"ondisk_log_size":2778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22729888,"num_objects":2775,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2776,"num_write_kb":22205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2833","reported_seq":2872,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.240296+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:41.240296+0000","last_peered":"2024-09-11T04:56:41.240296+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:41.240296+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2833,"log_dups_size":0,"ondisk_log_size":2833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23207936,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":22664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2850","reported_seq":2912,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.843089+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:40.843089+0000","last_peered":"2024-09-11T04:56:40.843089+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:40.843089+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23295136,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":527}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2867","reported_seq":2928,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.827166+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:40.827166+0000","last_peered":"2024-09-11T04:56:40.827166+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:40.827166+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23442592,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2899","reported_seq":2954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.204431+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:41.204431+0000","last_peered":"2024-09-11T04:56:41.204431+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:41.204431+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23677248,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":23145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2854","reported_seq":2930,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.688384+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:43.688384+0000","last_peered":"2024-09-11T04:56:43.688384+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:43.688384+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23360672,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":22814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2827","reported_seq":2881,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.162878+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:41.162878+0000","last_peered":"2024-09-11T04:56:41.162878+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:41.162878+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23113456,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":22580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2743","reported_seq":3199,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.219379+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:41.219379+0000","last_peered":"2024-09-11T04:56:41.219379+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:41.219379+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2743,"log_dups_size":0,"ondisk_log_size":2743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22461008,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":21935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2908","reported_seq":2981,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.651640+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:43.651640+0000","last_peered":"2024-09-11T04:56:43.651640+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:43.651640+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2908,"log_dups_size":0,"ondisk_log_size":2908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23812688,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2907,"num_write_kb":23255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2830","reported_seq":2879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:41.187692+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:41.187692+0000","last_peered":"2024-09-11T04:56:41.187692+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:41.187692+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2830,"log_dups_size":0,"ondisk_log_size":2830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23147680,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2828,"num_write_kb":22620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":539}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2893","reported_seq":2969,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.694866+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:43.694866+0000","last_peered":"2024-09-11T04:56:43.694866+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:43.694866+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2893,"log_dups_size":0,"ondisk_log_size":2893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23671968,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2891,"num_write_kb":23125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":572}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2836","reported_seq":2898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:40.709361+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:40.709361+0000","last_peered":"2024-09-11T04:56:40.709361+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:40.709361+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2836,"log_dups_size":0,"ondisk_log_size":2836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188640,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":22660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":370458816,"num_objects":45227,"num_object_clones":0,"num_object_copies":180908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45254,"num_write_kb":361914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":591400960,"data_stored":591400960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45281,"ondisk_log_size":45281,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315280,"kb_used_data":179456,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94056560,"statfs":{"total":96636764160,"available":96313917440,"internally_reserved":0,"allocated":183762944,"data_stored":183592134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,4,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327712,"kb_used_data":184080,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94044128,"statfs":{"total":96636764160,"available":96301187072,"internally_reserved":0,"allocated":188497920,"data_stored":188327110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109348,"kb_used_data":33300,"kb_used_omap":4,"kb_used_meta":76027,"kb_avail":94262492,"statfs":{"total":96636764160,"available":96524791808,"internally_reserved":0,"allocated":34099200,"data_stored":33928390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":77852370},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319720,"kb_used_data":179800,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94052120,"statfs":{"total":96636764160,"available":96309370880,"internally_reserved":0,"allocated":184115200,"data_stored":183944390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":183877632,"data_stored":183877632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34717696,"data_stored":34717696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":188370944,"data_stored":188370944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184434688,"data_stored":184434688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:46.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:56:46.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2802", 'reported_seq': 2843, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:41.233052+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:56:41.233052+0000', 'last_peered': '2024-09-11T04:56:41.233052+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:56:41.233052+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2802, 'log_dups_size': 0, 'ondisk_log_size': 2802, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22953984, 'num_objects': 2802, 'num_object_clones': 0, 'num_object_copies': 11208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2802, 'num_write_kb': 22416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 490}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:46.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:56:46.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2797", 'reported_seq': 2857, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:40.777787+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:56:40.777787+0000', 'last_peered': '2024-09-11T04:56:40.777787+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:56:40.777787+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2797, 'log_dups_size': 0, 'ondisk_log_size': 2797, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22877344, 'num_objects': 2793, 'num_object_clones': 0, 'num_object_copies': 11172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2795, 'num_write_kb': 22356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 514}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:46.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:56:46.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2771", 'reported_seq': 2845, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:43.691069+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:56:43.691069+0000', 'last_peered': '2024-09-11T04:56:43.691069+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:56:43.691069+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2771, 'log_dups_size': 0, 'ondisk_log_size': 2771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22674000, 'num_objects': 2768, 'num_object_clones': 0, 'num_object_copies': 11072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2768, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2770, 'num_write_kb': 22150, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 548}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:46.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2793", 'reported_seq': 2855, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:40.880240+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:56:40.880240+0000', 'last_peered': '2024-09-11T04:56:40.880240+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:56:40.880240+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2793, 'log_dups_size': 0, 'ondisk_log_size': 2793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22844576, 'num_objects': 2789, 'num_object_clones': 0, 'num_object_copies': 11156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2791, 'num_write_kb': 22317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2778", 'reported_seq': 2831, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:41.174553+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:56:41.174553+0000', 'last_peered': '2024-09-11T04:56:41.174553+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:56:41.174553+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2778, 'log_dups_size': 0, 'ondisk_log_size': 2778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22729888, 'num_objects': 2775, 'num_object_clones': 0, 'num_object_copies': 11100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2776, 'num_write_kb': 22205, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 525}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2833", 'reported_seq': 2872, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:41.240296+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:56:41.240296+0000', 'last_peered': '2024-09-11T04:56:41.240296+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:56:41.240296+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2833, 'log_dups_size': 0, 'ondisk_log_size': 2833, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23207936, 'num_objects': 2833, 'num_object_clones': 0, 'num_object_copies': 11332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2833, 'num_write_kb': 22664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 512}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2850", 'reported_seq': 2912, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:40.843089+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:56:40.843089+0000', 'last_peered': '2024-09-11T04:56:40.843089+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:56:40.843089+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2850, 'log_dups_size': 0, 'ondisk_log_size': 2850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23295136, 'num_objects': 2844, 'num_object_clones': 0, 'num_object_copies': 11376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2848, 'num_write_kb': 22764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 527}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2867", 'reported_seq': 2928, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:40.827166+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:56:40.827166+0000', 'last_peered': '2024-09-11T04:56:40.827166+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:56:40.827166+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2867, 'log_dups_size': 0, 'ondisk_log_size': 2867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23442592, 'num_objects': 2862, 'num_object_clones': 0, 'num_object_copies': 11448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2865, 'num_write_kb': 22908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:46.454 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2899", 'reported_seq': 2954, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:41.204431+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:56:41.204431+0000', 'last_peered': '2024-09-11T04:56:41.204431+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:56:41.204431+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2899, 'log_dups_size': 0, 'ondisk_log_size': 2899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23677248, 'num_objects': 2891, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2896, 'num_write_kb': 23145, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2854", 'reported_seq': 2930, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:43.688384+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:56:43.688384+0000', 'last_peered': '2024-09-11T04:56:43.688384+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:56:43.688384+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2854, 'log_dups_size': 0, 'ondisk_log_size': 2854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23360672, 'num_objects': 2852, 'num_object_clones': 0, 'num_object_copies': 11408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2852, 'num_write_kb': 22814, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 534}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2827", 'reported_seq': 2881, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:41.162878+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:56:41.162878+0000', 'last_peered': '2024-09-11T04:56:41.162878+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:56:41.162878+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2827, 'log_dups_size': 0, 'ondisk_log_size': 2827, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23113456, 'num_objects': 2822, 'num_object_clones': 0, 'num_object_copies': 11288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2824, 'num_write_kb': 22580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 499}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2743", 'reported_seq': 3199, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:41.219379+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:56:41.219379+0000', 'last_peered': '2024-09-11T04:56:41.219379+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:56:41.219379+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2743, 'log_dups_size': 0, 'ondisk_log_size': 2743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22461008, 'num_objects': 2742, 'num_object_clones': 0, 'num_object_copies': 10968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2742, 'num_write_kb': 21935, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 211, 'num_bytes_recovered': 1720320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2908", 'reported_seq': 2981, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:43.651640+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:56:43.651640+0000', 'last_peered': '2024-09-11T04:56:43.651640+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:56:43.651640+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2908, 'log_dups_size': 0, 'ondisk_log_size': 2908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23812688, 'num_objects': 2907, 'num_object_clones': 0, 'num_object_copies': 11628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2907, 'num_write_kb': 23255, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:46.455 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:56:46.456 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2830", 'reported_seq': 2879, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:41.187692+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:56:41.187692+0000', 'last_peered': '2024-09-11T04:56:41.187692+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:56:41.187692+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2830, 'log_dups_size': 0, 'ondisk_log_size': 2830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23147680, 'num_objects': 2826, 'num_object_clones': 0, 'num_object_copies': 11304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2828, 'num_write_kb': 22620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 539}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:46.456 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:56:46.456 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2893", 'reported_seq': 2969, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:43.694866+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:56:43.694866+0000', 'last_peered': '2024-09-11T04:56:43.694866+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:56:43.694866+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2893, 'log_dups_size': 0, 'ondisk_log_size': 2893, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23671968, 'num_objects': 2890, 'num_object_clones': 0, 'num_object_copies': 11560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2891, 'num_write_kb': 23125, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:46.456 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:56:46.456 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2836", 'reported_seq': 2898, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:40.709361+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:56:40.709361+0000', 'last_peered': '2024-09-11T04:56:40.709361+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:56:40.709361+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2836, 'log_dups_size': 0, 'ondisk_log_size': 2836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23188640, 'num_objects': 2831, 'num_object_clones': 0, 'num_object_copies': 11324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2834, 'num_write_kb': 22660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 520}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:46.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:56:46.695 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 63 20 47095 47075 5.83643 2.69531 0.0565537 0.0267517 2024-09-11T04:56:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:47.200 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36672/180908 objects degraded (20.271%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":134,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45227,"data_bytes":370458816,"bytes_used":1097789440,"bytes_avail":385449267200,"bytes_total":386547056640,"degraded_objects":36672,"degraded_total":180908,"degraded_ratio":0.20271077011519664,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":71645,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"write_bytes_sec":3017309,"read_op_per_sec":0,"write_op_per_sec":368},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-11T04:56:31.620416+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:56:47.221 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:56:47.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:47.695 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 64 20 47454 47434 5.78906 2.80469 0.0338297 0.026973 2024-09-11T04:56:47.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:47.857 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-11T04:56:47.627062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":382460096,"num_objects":46692,"num_object_clones":0,"num_object_copies":186768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46719,"num_write_kb":373634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46746,"ondisk_log_size":46746,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1157904,"kb_used_data":600016,"kb_used_omap":14,"kb_used_meta":557809,"kb_avail":376329456,"statfs":{"total":386547056640,"available":385361362944,"internally_reserved":0,"allocated":614416384,"data_stored":613733144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":571196982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,10,10,11,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44318720,"num_objects":5410,"num_object_clones":0,"num_object_copies":21640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":43280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005750"},"pg_stats":[{"pgid":"1.f","version":"31'2927","reported_seq":2968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.221376+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:46.221376+0000","last_peered":"2024-09-11T04:56:46.221376+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:46.221376+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2927,"log_dups_size":0,"ondisk_log_size":2927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23977984,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":23416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2926","reported_seq":2987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.841495+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:45.841495+0000","last_peered":"2024-09-11T04:56:45.841495+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:45.841495+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2926,"log_dups_size":0,"ondisk_log_size":2926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23934112,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":23388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2771","reported_seq":2845,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.691069+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:43.691069+0000","last_peered":"2024-09-11T04:56:43.691069+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:43.691069+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22674000,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2927","reported_seq":2989,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.824162+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:45.824162+0000","last_peered":"2024-09-11T04:56:45.824162+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:45.824162+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2927,"log_dups_size":0,"ondisk_log_size":2927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23942304,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2925,"num_write_kb":23389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":645}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2895","reported_seq":2947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.239051+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:46.239051+0000","last_peered":"2024-09-11T04:56:46.239051+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:46.239051+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2895,"log_dups_size":0,"ondisk_log_size":2895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23688352,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":23141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2960","reported_seq":3000,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.352150+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:46.352150+0000","last_peered":"2024-09-11T04:56:46.352150+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:46.352150+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2960,"log_dups_size":0,"ondisk_log_size":2960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":2960,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2960,"num_write_kb":23680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2968","reported_seq":3030,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.826645+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:45.826645+0000","last_peered":"2024-09-11T04:56:45.826645+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:45.826645+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2968,"log_dups_size":0,"ondisk_log_size":2968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24261792,"num_objects":2962,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2966,"num_write_kb":23708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":645}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2984","reported_seq":3045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.870329+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:45.870329+0000","last_peered":"2024-09-11T04:56:45.870329+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:45.870329+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24401056,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2982,"num_write_kb":23844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":618}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3011","reported_seq":3066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.209526+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:46.209526+0000","last_peered":"2024-09-11T04:56:46.209526+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:46.209526+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3011,"log_dups_size":0,"ondisk_log_size":3011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24594752,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3008,"num_write_kb":24041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2854","reported_seq":2930,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.688384+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:43.688384+0000","last_peered":"2024-09-11T04:56:43.688384+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:43.688384+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23360672,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":22814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2950","reported_seq":3005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.309325+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:46.309325+0000","last_peered":"2024-09-11T04:56:46.309325+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:46.309325+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24121072,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":23564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2845","reported_seq":3407,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.315446+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:46.315446+0000","last_peered":"2024-09-11T04:56:46.315446+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:46.315446+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2845,"log_dups_size":0,"ondisk_log_size":2845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23296592,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2844,"num_write_kb":22751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2908","reported_seq":2981,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.651640+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:43.651640+0000","last_peered":"2024-09-11T04:56:43.651640+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:43.651640+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2908,"log_dups_size":0,"ondisk_log_size":2908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23812688,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2907,"num_write_kb":23255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2961","reported_seq":3009,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.328276+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:46.328276+0000","last_peered":"2024-09-11T04:56:46.328276+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:46.328276+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2961,"log_dups_size":0,"ondisk_log_size":2961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24220832,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2959,"num_write_kb":23668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":670}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2893","reported_seq":2969,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:43.694866+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:43.694866+0000","last_peered":"2024-09-11T04:56:43.694866+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:43.694866+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2893,"log_dups_size":0,"ondisk_log_size":2893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23671968,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2891,"num_write_kb":23125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":572}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2966","reported_seq":3028,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.888670+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:45.888670+0000","last_peered":"2024-09-11T04:56:45.888670+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:45.888670+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2966,"log_dups_size":0,"ondisk_log_size":2966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24253600,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2964,"num_write_kb":23700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":382460096,"num_objects":46692,"num_object_clones":0,"num_object_copies":186768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46719,"num_write_kb":373634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":615460864,"data_stored":615460864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46746,"ondisk_log_size":46746,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349524,"kb_used_data":187332,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94022316,"statfs":{"total":96636764160,"available":96278851584,"internally_reserved":0,"allocated":191827968,"data_stored":191657158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":166063824},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327712,"kb_used_data":184080,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94044128,"statfs":{"total":96636764160,"available":96301187072,"internally_reserved":0,"allocated":188497920,"data_stored":188327110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126952,"kb_used_data":41176,"kb_used_omap":4,"kb_used_meta":85755,"kb_avail":94244888,"statfs":{"total":96636764160,"available":96506765312,"internally_reserved":0,"allocated":42164224,"data_stored":41993414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87813842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353716,"kb_used_data":187428,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94018124,"statfs":{"total":96636764160,"available":96274558976,"internally_reserved":0,"allocated":191926272,"data_stored":191755462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191758336,"data_stored":191758336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42942464,"data_stored":42942464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":188370944,"data_stored":188370944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192389120,"data_stored":192389120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:48.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 65 20 47824 47804 5.74447 2.89062 0.0555819 0.0271833 2024-09-11T04:56:49.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 66 20 48224 48204 5.70478 3.125 0.0518617 0.0273709 2024-09-11T04:56:50.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 67 20 48623 48603 5.66616 3.11719 0.0405892 0.0275627 2024-09-11T04:56:50.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:51.437 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:51.438 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:51.473 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-11T04:56:49.627516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":386302144,"num_objects":47161,"num_object_clones":0,"num_object_copies":188644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47188,"num_write_kb":377386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47215,"ondisk_log_size":47215,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191744,"kb_used_data":607484,"kb_used_omap":14,"kb_used_meta":584177,"kb_avail":376295616,"statfs":{"total":386547056640,"available":385326710784,"internally_reserved":0,"allocated":622063616,"data_stored":621380376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":598197814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,13,18,16,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35831808,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":34992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005753"},"pg_stats":[{"pgid":"1.f","version":"31'2927","reported_seq":2968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.221376+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:46.221376+0000","last_peered":"2024-09-11T04:56:46.221376+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:46.221376+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2927,"log_dups_size":0,"ondisk_log_size":2927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23977984,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":23416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2926","reported_seq":2987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.841495+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:45.841495+0000","last_peered":"2024-09-11T04:56:45.841495+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:45.841495+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2926,"log_dups_size":0,"ondisk_log_size":2926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23934112,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":23388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2871","reported_seq":2945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.728141+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:48.728141+0000","last_peered":"2024-09-11T04:56:48.728141+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:48.728141+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2871,"log_dups_size":0,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23493200,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":22950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2927","reported_seq":2989,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.824162+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:45.824162+0000","last_peered":"2024-09-11T04:56:45.824162+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:45.824162+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2927,"log_dups_size":0,"ondisk_log_size":2927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23942304,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2925,"num_write_kb":23389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":645}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2895","reported_seq":2947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.239051+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:46.239051+0000","last_peered":"2024-09-11T04:56:46.239051+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:46.239051+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2895,"log_dups_size":0,"ondisk_log_size":2895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23688352,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":23141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2960","reported_seq":3000,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.352150+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:46.352150+0000","last_peered":"2024-09-11T04:56:46.352150+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:46.352150+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2960,"log_dups_size":0,"ondisk_log_size":2960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":2960,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2960,"num_write_kb":23680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2968","reported_seq":3030,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.826645+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:45.826645+0000","last_peered":"2024-09-11T04:56:45.826645+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:45.826645+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2968,"log_dups_size":0,"ondisk_log_size":2968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24261792,"num_objects":2962,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2966,"num_write_kb":23708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":645}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2984","reported_seq":3045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.870329+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:45.870329+0000","last_peered":"2024-09-11T04:56:45.870329+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:45.870329+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24401056,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2982,"num_write_kb":23844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":618}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3011","reported_seq":3066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.209526+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:46.209526+0000","last_peered":"2024-09-11T04:56:46.209526+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:46.209526+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3011,"log_dups_size":0,"ondisk_log_size":3011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24594752,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3008,"num_write_kb":24041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2978","reported_seq":3054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.712195+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:48.712195+0000","last_peered":"2024-09-11T04:56:48.712195+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:48.712195+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24376480,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2950","reported_seq":3005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.309325+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:46.309325+0000","last_peered":"2024-09-11T04:56:46.309325+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:46.309325+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24121072,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":23564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2845","reported_seq":3407,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.315446+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:46.315446+0000","last_peered":"2024-09-11T04:56:46.315446+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:46.315446+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2845,"log_dups_size":0,"ondisk_log_size":2845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23296592,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2844,"num_write_kb":22751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3027","reported_seq":3100,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.662826+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:48.662826+0000","last_peered":"2024-09-11T04:56:48.662826+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:48.662826+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3027,"log_dups_size":0,"ondisk_log_size":3027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24787536,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":24207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":694}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2961","reported_seq":3009,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:46.328276+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:46.328276+0000","last_peered":"2024-09-11T04:56:46.328276+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:46.328276+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2961,"log_dups_size":0,"ondisk_log_size":2961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24220832,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2959,"num_write_kb":23668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":670}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3019","reported_seq":3096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.737229+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:48.737229+0000","last_peered":"2024-09-11T04:56:48.737229+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:48.737229+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3019,"log_dups_size":0,"ondisk_log_size":3019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24704160,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2966","reported_seq":3028,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:45.888670+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:45.888670+0000","last_peered":"2024-09-11T04:56:45.888670+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:45.888670+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2966,"log_dups_size":0,"ondisk_log_size":2966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24253600,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2964,"num_write_kb":23700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":386302144,"num_objects":47161,"num_object_clones":0,"num_object_copies":188644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47188,"num_write_kb":377386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":623054848,"data_stored":623054848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47215,"ondisk_log_size":47215,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349524,"kb_used_data":187332,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94022316,"statfs":{"total":96636764160,"available":96278851584,"internally_reserved":0,"allocated":191827968,"data_stored":191657158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":166063824},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361552,"kb_used_data":191548,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":94010288,"statfs":{"total":96636764160,"available":96266534912,"internally_reserved":0,"allocated":196145152,"data_stored":195974342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126952,"kb_used_data":41176,"kb_used_omap":4,"kb_used_meta":85755,"kb_avail":94244888,"statfs":{"total":96636764160,"available":96506765312,"internally_reserved":0,"allocated":42164224,"data_stored":41993414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87813842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353716,"kb_used_data":187428,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94018124,"statfs":{"total":96636764160,"available":96274558976,"internally_reserved":0,"allocated":191926272,"data_stored":191755462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191758336,"data_stored":191758336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42942464,"data_stored":42942464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195964928,"data_stored":195964928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192389120,"data_stored":192389120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:51.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:51.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 68 20 48981 48961 5.62397 2.79688 0.15573 0.0277684 2024-09-11T04:56:52.065 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:52.066 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:52.097 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-11T04:56:51.627970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397828288,"num_objects":48568,"num_object_clones":0,"num_object_copies":194272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48595,"num_write_kb":388642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48622,"ondisk_log_size":48622,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1215112,"kb_used_data":621124,"kb_used_omap":14,"kb_used_meta":593905,"kb_avail":376272248,"statfs":{"total":386547056640,"available":385302781952,"internally_reserved":0,"allocated":636030976,"data_stored":635347736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":608159286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,11,18,16,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43245568,"num_objects":5279,"num_object_clones":0,"num_object_copies":21116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005689"},"pg_stats":[{"pgid":"1.f","version":"31'3035","reported_seq":3076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.086324+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:51.086324+0000","last_peered":"2024-09-11T04:56:51.086324+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:51.086324+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24862720,"num_objects":3035,"num_object_clones":0,"num_object_copies":12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3035,"num_write_kb":24280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3050","reported_seq":3110,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.943388+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:50.943388+0000","last_peered":"2024-09-11T04:56:50.943388+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:50.943388+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3050,"log_dups_size":0,"ondisk_log_size":3050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24949920,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":24380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2871","reported_seq":2945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.728141+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:48.728141+0000","last_peered":"2024-09-11T04:56:48.728141+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:48.728141+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2871,"log_dups_size":0,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23493200,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":22950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3036","reported_seq":3098,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.816260+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:50.816260+0000","last_peered":"2024-09-11T04:56:50.816260+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:50.816260+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3036,"log_dups_size":0,"ondisk_log_size":3036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3034,"num_write_kb":24261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3029","reported_seq":3082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.313999+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:51.313999+0000","last_peered":"2024-09-11T04:56:51.313999+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:51.313999+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3029,"log_dups_size":0,"ondisk_log_size":3029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24786080,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":24213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3069","reported_seq":3109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.308326+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:51.308326+0000","last_peered":"2024-09-11T04:56:51.308326+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:51.308326+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3069,"log_dups_size":0,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":24552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3071","reported_seq":3133,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.892037+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:50.892037+0000","last_peered":"2024-09-11T04:56:50.892037+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:50.892037+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25105568,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":24532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3115","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.838613+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:50.838613+0000","last_peered":"2024-09-11T04:56:50.838613+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:50.838613+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25474208,"num_objects":3110,"num_object_clones":0,"num_object_copies":12440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3142","reported_seq":3197,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.302560+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:51.302560+0000","last_peered":"2024-09-11T04:56:51.302560+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:51.302560+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25667904,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3139,"num_write_kb":25089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2978","reported_seq":3054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.712195+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:48.712195+0000","last_peered":"2024-09-11T04:56:48.712195+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:48.712195+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24376480,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3051","reported_seq":3106,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.267703+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:51.267703+0000","last_peered":"2024-09-11T04:56:51.267703+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:51.267703+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24948464,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":24372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2976","reported_seq":3650,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.297067+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:51.297067+0000","last_peered":"2024-09-11T04:56:51.297067+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:51.297067+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2976,"log_dups_size":0,"ondisk_log_size":2976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24369744,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3027","reported_seq":3100,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.662826+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:48.662826+0000","last_peered":"2024-09-11T04:56:48.662826+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:48.662826+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3027,"log_dups_size":0,"ondisk_log_size":3027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24787536,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":24207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":694}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3068","reported_seq":3117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.265977+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:51.265977+0000","last_peered":"2024-09-11T04:56:51.265977+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:51.265977+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3068,"log_dups_size":0,"ondisk_log_size":3068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25097376,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3066,"num_write_kb":24524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":777}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3019","reported_seq":3096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.737229+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:48.737229+0000","last_peered":"2024-09-11T04:56:48.737229+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:48.737229+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3019,"log_dups_size":0,"ondisk_log_size":3019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24704160,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3085","reported_seq":3145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.934781+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:50.934781+0000","last_peered":"2024-09-11T04:56:50.934781+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:50.934781+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":397828288,"num_objects":48568,"num_object_clones":0,"num_object_copies":194272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48595,"num_write_kb":388642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638607360,"data_stored":638607360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48622,"ondisk_log_size":48622,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379304,"kb_used_data":194904,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93992536,"statfs":{"total":96636764160,"available":96248356864,"internally_reserved":0,"allocated":199581696,"data_stored":199410886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,7,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361552,"kb_used_data":191548,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":94010288,"statfs":{"total":96636764160,"available":96266534912,"internally_reserved":0,"allocated":196145152,"data_stored":195974342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126952,"kb_used_data":41176,"kb_used_omap":4,"kb_used_meta":85755,"kb_avail":94244888,"statfs":{"total":96636764160,"available":96506765312,"internally_reserved":0,"allocated":42164224,"data_stored":41993414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87813842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347304,"kb_used_data":193496,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94024536,"statfs":{"total":96636764160,"available":96281124864,"internally_reserved":0,"allocated":198139904,"data_stored":197969094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199532544,"data_stored":199532544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42942464,"data_stored":42942464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195964928,"data_stored":195964928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200167424,"data_stored":200167424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3035", 'reported_seq': 3076, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:51.086324+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:56:51.086324+0000', 'last_peered': '2024-09-11T04:56:51.086324+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:56:51.086324+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3035, 'log_dups_size': 0, 'ondisk_log_size': 3035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24862720, 'num_objects': 3035, 'num_object_clones': 0, 'num_object_copies': 12140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3035, 'num_write_kb': 24280, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 723}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3050", 'reported_seq': 3110, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:50.943388+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:56:50.943388+0000', 'last_peered': '2024-09-11T04:56:50.943388+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:56:50.943388+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3050, 'log_dups_size': 0, 'ondisk_log_size': 3050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24949920, 'num_objects': 3046, 'num_object_clones': 0, 'num_object_copies': 12184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3048, 'num_write_kb': 24380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2871", 'reported_seq': 2945, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:48.728141+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:56:48.728141+0000', 'last_peered': '2024-09-11T04:56:48.728141+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:56:48.728141+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2871, 'log_dups_size': 0, 'ondisk_log_size': 2871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23493200, 'num_objects': 2868, 'num_object_clones': 0, 'num_object_copies': 11472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2870, 'num_write_kb': 22950, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 648}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3036", 'reported_seq': 3098, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:50.816260+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:56:50.816260+0000', 'last_peered': '2024-09-11T04:56:50.816260+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:56:50.816260+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3036, 'log_dups_size': 0, 'ondisk_log_size': 3036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24835232, 'num_objects': 3032, 'num_object_clones': 0, 'num_object_copies': 12128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3034, 'num_write_kb': 24261, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:56:52.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3029", 'reported_seq': 3082, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:51.313999+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:56:51.313999+0000', 'last_peered': '2024-09-11T04:56:51.313999+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:56:51.313999+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3029, 'log_dups_size': 0, 'ondisk_log_size': 3029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24786080, 'num_objects': 3026, 'num_object_clones': 0, 'num_object_copies': 12104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3027, 'num_write_kb': 24213, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 776}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3069", 'reported_seq': 3109, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:51.308326+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:56:51.308326+0000', 'last_peered': '2024-09-11T04:56:51.308326+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:56:51.308326+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3069, 'log_dups_size': 0, 'ondisk_log_size': 3069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25141248, 'num_objects': 3069, 'num_object_clones': 0, 'num_object_copies': 12276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3069, 'num_write_kb': 24552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 748}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3071", 'reported_seq': 3133, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:50.892037+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:56:50.892037+0000', 'last_peered': '2024-09-11T04:56:50.892037+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:56:50.892037+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3071, 'log_dups_size': 0, 'ondisk_log_size': 3071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25105568, 'num_objects': 3065, 'num_object_clones': 0, 'num_object_copies': 12260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3069, 'num_write_kb': 24532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 748}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3115", 'reported_seq': 3176, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:50.838613+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:56:50.838613+0000', 'last_peered': '2024-09-11T04:56:50.838613+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:56:50.838613+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3115, 'log_dups_size': 0, 'ondisk_log_size': 3115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25474208, 'num_objects': 3110, 'num_object_clones': 0, 'num_object_copies': 12440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3113, 'num_write_kb': 24892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3142", 'reported_seq': 3197, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:51.302560+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:56:51.302560+0000', 'last_peered': '2024-09-11T04:56:51.302560+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:56:51.302560+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3142, 'log_dups_size': 0, 'ondisk_log_size': 3142, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25667904, 'num_objects': 3134, 'num_object_clones': 0, 'num_object_copies': 12536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3139, 'num_write_kb': 25089, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 740}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:56:52.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2978", 'reported_seq': 3054, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:48.712195+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:56:48.712195+0000', 'last_peered': '2024-09-11T04:56:48.712195+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:56:48.712195+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2978, 'log_dups_size': 0, 'ondisk_log_size': 2978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24376480, 'num_objects': 2976, 'num_object_clones': 0, 'num_object_copies': 11904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2976, 'num_write_kb': 23806, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 658}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3051", 'reported_seq': 3106, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:51.267703+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:56:51.267703+0000', 'last_peered': '2024-09-11T04:56:51.267703+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:56:51.267703+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3051, 'log_dups_size': 0, 'ondisk_log_size': 3051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24948464, 'num_objects': 3046, 'num_object_clones': 0, 'num_object_copies': 12184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3048, 'num_write_kb': 24372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 723}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2976", 'reported_seq': 3650, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:51.297067+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:56:51.297067+0000', 'last_peered': '2024-09-11T04:56:51.297067+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:56:51.297067+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2976, 'log_dups_size': 0, 'ondisk_log_size': 2976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24369744, 'num_objects': 2975, 'num_object_clones': 0, 'num_object_copies': 11900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2975, 'num_write_kb': 23799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 320, 'num_bytes_recovered': 2613248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3027", 'reported_seq': 3100, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:48.662826+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:56:48.662826+0000', 'last_peered': '2024-09-11T04:56:48.662826+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:56:48.662826+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3027, 'log_dups_size': 0, 'ondisk_log_size': 3027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24787536, 'num_objects': 3026, 'num_object_clones': 0, 'num_object_copies': 12104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3026, 'num_write_kb': 24207, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 694}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3068", 'reported_seq': 3117, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:51.265977+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:56:51.265977+0000', 'last_peered': '2024-09-11T04:56:51.265977+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:56:51.265977+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3068, 'log_dups_size': 0, 'ondisk_log_size': 3068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25097376, 'num_objects': 3064, 'num_object_clones': 0, 'num_object_copies': 12256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3066, 'num_write_kb': 24524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 777}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3019", 'reported_seq': 3096, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:48.737229+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:56:48.737229+0000', 'last_peered': '2024-09-11T04:56:48.737229+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:56:48.737229+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3019, 'log_dups_size': 0, 'ondisk_log_size': 3019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24704160, 'num_objects': 3016, 'num_object_clones': 0, 'num_object_copies': 12064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3017, 'num_write_kb': 24133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 698}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:52.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:56:52.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3085", 'reported_seq': 3145, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:50.934781+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:56:50.934781+0000', 'last_peered': '2024-09-11T04:56:50.934781+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:56:50.934781+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3085, 'log_dups_size': 0, 'ondisk_log_size': 3085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25228448, 'num_objects': 3080, 'num_object_clones': 0, 'num_object_copies': 12320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3083, 'num_write_kb': 24652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 769}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:52.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:56:52.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 69 20 49332 49312 5.5822 2.74219 0.0722292 0.0279726 2024-09-11T04:56:52.951 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:52.952 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36563/194272 objects degraded (18.821%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":139,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":48568,"data_bytes":397828288,"bytes_used":1244274688,"bytes_avail":385302781952,"bytes_total":386547056640,"degraded_objects":36563,"degraded_total":194272,"degraded_ratio":0.18820519683742382,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":111904,"recovering_keys_per_sec":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"write_bytes_sec":3602089,"read_op_per_sec":0,"write_op_per_sec":439},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-11T04:56:31.620416+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:56:52.974 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:56:52.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:53.563 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:53.594 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-11T04:56:51.627970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397828288,"num_objects":48568,"num_object_clones":0,"num_object_copies":194272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48595,"num_write_kb":388642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48622,"ondisk_log_size":48622,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1215112,"kb_used_data":621124,"kb_used_omap":14,"kb_used_meta":593905,"kb_avail":376272248,"statfs":{"total":386547056640,"available":385302781952,"internally_reserved":0,"allocated":636030976,"data_stored":635347736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":608159286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,11,18,16,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43245568,"num_objects":5279,"num_object_clones":0,"num_object_copies":21116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005689"},"pg_stats":[{"pgid":"1.f","version":"31'3035","reported_seq":3076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.086324+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:51.086324+0000","last_peered":"2024-09-11T04:56:51.086324+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:51.086324+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24862720,"num_objects":3035,"num_object_clones":0,"num_object_copies":12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3035,"num_write_kb":24280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3050","reported_seq":3110,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.943388+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:50.943388+0000","last_peered":"2024-09-11T04:56:50.943388+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:50.943388+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3050,"log_dups_size":0,"ondisk_log_size":3050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24949920,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":24380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2871","reported_seq":2945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.728141+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:48.728141+0000","last_peered":"2024-09-11T04:56:48.728141+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:48.728141+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2871,"log_dups_size":0,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23493200,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":22950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3036","reported_seq":3098,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.816260+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:50.816260+0000","last_peered":"2024-09-11T04:56:50.816260+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:50.816260+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3036,"log_dups_size":0,"ondisk_log_size":3036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3034,"num_write_kb":24261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3029","reported_seq":3082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.313999+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:51.313999+0000","last_peered":"2024-09-11T04:56:51.313999+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:51.313999+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3029,"log_dups_size":0,"ondisk_log_size":3029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24786080,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":24213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3069","reported_seq":3109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.308326+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:51.308326+0000","last_peered":"2024-09-11T04:56:51.308326+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:51.308326+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3069,"log_dups_size":0,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":24552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3071","reported_seq":3133,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.892037+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:50.892037+0000","last_peered":"2024-09-11T04:56:50.892037+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:50.892037+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25105568,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":24532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3115","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.838613+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:50.838613+0000","last_peered":"2024-09-11T04:56:50.838613+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:50.838613+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25474208,"num_objects":3110,"num_object_clones":0,"num_object_copies":12440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3142","reported_seq":3197,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.302560+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:51.302560+0000","last_peered":"2024-09-11T04:56:51.302560+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:51.302560+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25667904,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3139,"num_write_kb":25089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2978","reported_seq":3054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.712195+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:48.712195+0000","last_peered":"2024-09-11T04:56:48.712195+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:48.712195+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24376480,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3051","reported_seq":3106,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.267703+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:51.267703+0000","last_peered":"2024-09-11T04:56:51.267703+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:51.267703+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24948464,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":24372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2976","reported_seq":3650,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.297067+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:51.297067+0000","last_peered":"2024-09-11T04:56:51.297067+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:51.297067+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2976,"log_dups_size":0,"ondisk_log_size":2976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24369744,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3027","reported_seq":3100,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.662826+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:48.662826+0000","last_peered":"2024-09-11T04:56:48.662826+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:48.662826+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3027,"log_dups_size":0,"ondisk_log_size":3027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24787536,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":24207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":694}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3068","reported_seq":3117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.265977+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:51.265977+0000","last_peered":"2024-09-11T04:56:51.265977+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:51.265977+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3068,"log_dups_size":0,"ondisk_log_size":3068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25097376,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3066,"num_write_kb":24524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":777}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3019","reported_seq":3096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:48.737229+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:48.737229+0000","last_peered":"2024-09-11T04:56:48.737229+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:48.737229+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3019,"log_dups_size":0,"ondisk_log_size":3019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24704160,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3085","reported_seq":3145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.934781+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:50.934781+0000","last_peered":"2024-09-11T04:56:50.934781+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:50.934781+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":397828288,"num_objects":48568,"num_object_clones":0,"num_object_copies":194272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48595,"num_write_kb":388642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638607360,"data_stored":638607360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48622,"ondisk_log_size":48622,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379304,"kb_used_data":194904,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93992536,"statfs":{"total":96636764160,"available":96248356864,"internally_reserved":0,"allocated":199581696,"data_stored":199410886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,7,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361552,"kb_used_data":191548,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":94010288,"statfs":{"total":96636764160,"available":96266534912,"internally_reserved":0,"allocated":196145152,"data_stored":195974342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126952,"kb_used_data":41176,"kb_used_omap":4,"kb_used_meta":85755,"kb_avail":94244888,"statfs":{"total":96636764160,"available":96506765312,"internally_reserved":0,"allocated":42164224,"data_stored":41993414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87813842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347304,"kb_used_data":193496,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94024536,"statfs":{"total":96636764160,"available":96281124864,"internally_reserved":0,"allocated":198139904,"data_stored":197969094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199532544,"data_stored":199532544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42942464,"data_stored":42942464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195964928,"data_stored":195964928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200167424,"data_stored":200167424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:53.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 70 20 49680 49660 5.5413 2.71875 0.0479062 0.0281823 2024-09-11T04:56:54.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 71 20 50078 50058 5.50705 3.10938 0.0598289 0.0283586 2024-09-11T04:56:55.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 72 20 50470 50450 5.4731 3.0625 0.0308012 0.0285325 2024-09-11T04:56:56.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:56.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 73 20 50860 50840 5.43986 3.04688 0.0599999 0.0287061 2024-09-11T04:56:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:57.214 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-11T04:56:55.628904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":401588416,"num_objects":49027,"num_object_clones":0,"num_object_copies":196108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49054,"num_write_kb":392314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49081,"ondisk_log_size":49081,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1244088,"kb_used_data":636216,"kb_used_omap":14,"kb_used_meta":607793,"kb_avail":376243272,"statfs":{"total":386547056640,"available":385273110528,"internally_reserved":0,"allocated":651485184,"data_stored":650801944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":622380598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,13,14,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34971648,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4269,"num_write_kb":34152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":892928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002699"},"pg_stats":[{"pgid":"1.f","version":"31'3035","reported_seq":3076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.086324+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:51.086324+0000","last_peered":"2024-09-11T04:56:51.086324+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:51.086324+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24862720,"num_objects":3035,"num_object_clones":0,"num_object_copies":12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3035,"num_write_kb":24280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3050","reported_seq":3110,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.943388+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:50.943388+0000","last_peered":"2024-09-11T04:56:50.943388+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:50.943388+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3050,"log_dups_size":0,"ondisk_log_size":3050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24949920,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":24380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2969","reported_seq":3043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.721238+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:53.721238+0000","last_peered":"2024-09-11T04:56:53.721238+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:53.721238+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2969,"log_dups_size":0,"ondisk_log_size":2969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24296016,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":23734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3036","reported_seq":3098,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.816260+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:50.816260+0000","last_peered":"2024-09-11T04:56:50.816260+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:50.816260+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3036,"log_dups_size":0,"ondisk_log_size":3036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24835232,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3034,"num_write_kb":24261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3029","reported_seq":3082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.313999+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:51.313999+0000","last_peered":"2024-09-11T04:56:51.313999+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:51.313999+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3029,"log_dups_size":0,"ondisk_log_size":3029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24786080,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":24213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3069","reported_seq":3109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.308326+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:51.308326+0000","last_peered":"2024-09-11T04:56:51.308326+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:51.308326+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3069,"log_dups_size":0,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":24552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3071","reported_seq":3133,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.892037+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:50.892037+0000","last_peered":"2024-09-11T04:56:50.892037+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:50.892037+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25105568,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":24532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3115","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.838613+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:50.838613+0000","last_peered":"2024-09-11T04:56:50.838613+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:50.838613+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25474208,"num_objects":3110,"num_object_clones":0,"num_object_copies":12440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3142","reported_seq":3197,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.302560+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:51.302560+0000","last_peered":"2024-09-11T04:56:51.302560+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:51.302560+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25667904,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3139,"num_write_kb":25089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3103","reported_seq":3178,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.686533+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:53.686533+0000","last_peered":"2024-09-11T04:56:53.686533+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:53.686533+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400480,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":24806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3051","reported_seq":3106,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.267703+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:51.267703+0000","last_peered":"2024-09-11T04:56:51.267703+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:51.267703+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24948464,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":24372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2976","reported_seq":3650,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.297067+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:51.297067+0000","last_peered":"2024-09-11T04:56:51.297067+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:51.297067+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2976,"log_dups_size":0,"ondisk_log_size":2976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24369744,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3148","reported_seq":3221,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.634385+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:53.634385+0000","last_peered":"2024-09-11T04:56:53.634385+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:53.634385+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25778768,"num_objects":3147,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3147,"num_write_kb":25175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3068","reported_seq":3117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:51.265977+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:51.265977+0000","last_peered":"2024-09-11T04:56:51.265977+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:51.265977+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3068,"log_dups_size":0,"ondisk_log_size":3068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25097376,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3066,"num_write_kb":24524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":777}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3134","reported_seq":3211,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.699037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:53.699037+0000","last_peered":"2024-09-11T04:56:53.699037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:53.699037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25646240,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3132,"num_write_kb":25053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3085","reported_seq":3145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:50.934781+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:50.934781+0000","last_peered":"2024-09-11T04:56:50.934781+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:50.934781+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228448,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":769}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":401588416,"num_objects":49027,"num_object_clones":0,"num_object_copies":196108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49054,"num_write_kb":392314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":2613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":654266368,"data_stored":654266368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49081,"ondisk_log_size":49081,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379304,"kb_used_data":194904,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93992536,"statfs":{"total":96636764160,"available":96248356864,"internally_reserved":0,"allocated":199581696,"data_stored":199410886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,7,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374392,"kb_used_data":198824,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93997448,"statfs":{"total":96636764160,"available":96253386752,"internally_reserved":0,"allocated":203595776,"data_stored":203424966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143088,"kb_used_data":48992,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94228752,"statfs":{"total":96636764160,"available":96490242048,"internally_reserved":0,"allocated":50167808,"data_stored":49996998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96333522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347304,"kb_used_data":193496,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94024536,"statfs":{"total":96636764160,"available":96281124864,"internally_reserved":0,"allocated":198139904,"data_stored":197969094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199532544,"data_stored":199532544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51003392,"data_stored":51003392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203563008,"data_stored":203563008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200167424,"data_stored":200167424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:57.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:57.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 74 20 51208 51188 5.40309 2.71875 0.107634 0.0288997 2024-09-11T04:56:57.845 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:57.845 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:57.866 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-11T04:56:57.629368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":413262016,"num_objects":50452,"num_object_clones":0,"num_object_copies":201808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50479,"num_write_kb":403714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50506,"ondisk_log_size":50506,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1263780,"kb_used_data":658660,"kb_used_omap":14,"kb_used_meta":605041,"kb_avail":376223580,"statfs":{"total":386547056640,"available":385252945920,"internally_reserved":0,"allocated":674467840,"data_stored":673784600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":619562550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,10,16,9,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42803200,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":41800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"1.f","version":"31'3153","reported_seq":3194,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.409544+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:56.409544+0000","last_peered":"2024-09-11T04:56:56.409544+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:56.409544+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3153,"log_dups_size":0,"ondisk_log_size":3153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25829376,"num_objects":3153,"num_object_clones":0,"num_object_copies":12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3159","reported_seq":3219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.945457+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:55.945457+0000","last_peered":"2024-09-11T04:56:55.945457+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:55.945457+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25842848,"num_objects":3155,"num_object_clones":0,"num_object_copies":12620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3157,"num_write_kb":25252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2969","reported_seq":3043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.721238+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:53.721238+0000","last_peered":"2024-09-11T04:56:53.721238+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:53.721238+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2969,"log_dups_size":0,"ondisk_log_size":2969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24296016,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":23734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3124","reported_seq":3186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.950560+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:55.950560+0000","last_peered":"2024-09-11T04:56:55.950560+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:55.950560+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3124,"log_dups_size":0,"ondisk_log_size":3124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25556128,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3122,"num_write_kb":24965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3167","reported_seq":3218,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.367948+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:56.367948+0000","last_peered":"2024-09-11T04:56:56.367948+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:56.367948+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25916576,"num_objects":3164,"num_object_clones":0,"num_object_copies":12656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3165,"num_write_kb":25317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3173","reported_seq":3213,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.302416+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:56.302416+0000","last_peered":"2024-09-11T04:56:56.302416+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:56.302416+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25993216,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3173,"num_write_kb":25384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3198","reported_seq":3260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.940178+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:55.940178+0000","last_peered":"2024-09-11T04:56:55.940178+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:55.940178+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3198,"log_dups_size":0,"ondisk_log_size":3198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26145952,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":25548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":875}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3234","reported_seq":3295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.901132+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:55.901132+0000","last_peered":"2024-09-11T04:56:55.901132+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:55.901132+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3234,"log_dups_size":0,"ondisk_log_size":3234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26449056,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3232,"num_write_kb":25844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3275","reported_seq":3330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.370011+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:56.370011+0000","last_peered":"2024-09-11T04:56:56.370011+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:56.370011+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3275,"log_dups_size":0,"ondisk_log_size":3275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26757440,"num_objects":3267,"num_object_clones":0,"num_object_copies":13068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3272,"num_write_kb":26153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3103","reported_seq":3178,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.686533+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:53.686533+0000","last_peered":"2024-09-11T04:56:53.686533+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:53.686533+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400480,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":24806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3172","reported_seq":3227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.367494+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:56.367494+0000","last_peered":"2024-09-11T04:56:56.367494+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:56.367494+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25939696,"num_objects":3167,"num_object_clones":0,"num_object_copies":12668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3169,"num_write_kb":25340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3107","reported_seq":3871,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.413611+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:56.413611+0000","last_peered":"2024-09-11T04:56:56.413611+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:56.413611+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25442896,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3106,"num_write_kb":24847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1235}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3148","reported_seq":3221,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.634385+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:53.634385+0000","last_peered":"2024-09-11T04:56:53.634385+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:53.634385+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25778768,"num_objects":3147,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3147,"num_write_kb":25175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3193","reported_seq":3242,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.312840+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:56.312840+0000","last_peered":"2024-09-11T04:56:56.312840+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:56.312840+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3193,"log_dups_size":0,"ondisk_log_size":3193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26121376,"num_objects":3189,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3191,"num_write_kb":25524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3134","reported_seq":3211,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.699037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:53.699037+0000","last_peered":"2024-09-11T04:56:53.699037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:53.699037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25646240,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3132,"num_write_kb":25053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3197","reported_seq":3258,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.880856+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:55.880856+0000","last_peered":"2024-09-11T04:56:55.880856+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:55.880856+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3197,"log_dups_size":0,"ondisk_log_size":3197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26145952,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":25548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":413262016,"num_objects":50452,"num_object_clones":0,"num_object_copies":201808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50479,"num_write_kb":403714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":677593088,"data_stored":677593088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50506,"ondisk_log_size":50506,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374452,"kb_used_data":202532,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93997388,"statfs":{"total":96636764160,"available":96253325312,"internally_reserved":0,"allocated":207392768,"data_stored":207221958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,8,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374392,"kb_used_data":198824,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93997448,"statfs":{"total":96636764160,"available":96253386752,"internally_reserved":0,"allocated":203595776,"data_stored":203424966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138016,"kb_used_data":56400,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94233824,"statfs":{"total":96636764160,"available":96495435776,"internally_reserved":0,"allocated":57753600,"data_stored":57582790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":83554002},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":376920,"kb_used_data":200904,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93994920,"statfs":{"total":96636764160,"available":96250798080,"internally_reserved":0,"allocated":205725696,"data_stored":205554886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207171584,"data_stored":207171584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58941440,"data_stored":58941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203563008,"data_stored":203563008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207917056,"data_stored":207917056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:57.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:56:57.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3153", 'reported_seq': 3194, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:56.409544+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:56:56.409544+0000', 'last_peered': '2024-09-11T04:56:56.409544+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:56:56.409544+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3153, 'log_dups_size': 0, 'ondisk_log_size': 3153, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25829376, 'num_objects': 3153, 'num_object_clones': 0, 'num_object_copies': 12612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3153, 'num_write_kb': 25224, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 841}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:57.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:56:57.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3159", 'reported_seq': 3219, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:55.945457+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:56:55.945457+0000', 'last_peered': '2024-09-11T04:56:55.945457+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:56:55.945457+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3159, 'log_dups_size': 0, 'ondisk_log_size': 3159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25842848, 'num_objects': 3155, 'num_object_clones': 0, 'num_object_copies': 12620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3157, 'num_write_kb': 25252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2969", 'reported_seq': 3043, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:53.721238+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:56:53.721238+0000', 'last_peered': '2024-09-11T04:56:53.721238+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:56:53.721238+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 2969, 'log_dups_size': 0, 'ondisk_log_size': 2969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24296016, 'num_objects': 2966, 'num_object_clones': 0, 'num_object_copies': 11864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2968, 'num_write_kb': 23734, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 746}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3124", 'reported_seq': 3186, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:55.950560+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:56:55.950560+0000', 'last_peered': '2024-09-11T04:56:55.950560+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:56:55.950560+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3124, 'log_dups_size': 0, 'ondisk_log_size': 3124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25556128, 'num_objects': 3120, 'num_object_clones': 0, 'num_object_copies': 12480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3122, 'num_write_kb': 24965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 842}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3167", 'reported_seq': 3218, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:56.367948+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:56:56.367948+0000', 'last_peered': '2024-09-11T04:56:56.367948+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:56:56.367948+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3167, 'log_dups_size': 0, 'ondisk_log_size': 3167, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25916576, 'num_objects': 3164, 'num_object_clones': 0, 'num_object_copies': 12656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3165, 'num_write_kb': 25317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 914}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3173", 'reported_seq': 3213, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:56.302416+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:56:56.302416+0000', 'last_peered': '2024-09-11T04:56:56.302416+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:56:56.302416+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3173, 'log_dups_size': 0, 'ondisk_log_size': 3173, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25993216, 'num_objects': 3173, 'num_object_clones': 0, 'num_object_copies': 12692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3173, 'num_write_kb': 25384, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 852}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3198", 'reported_seq': 3260, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:55.940178+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:56:55.940178+0000', 'last_peered': '2024-09-11T04:56:55.940178+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:56:55.940178+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3198, 'log_dups_size': 0, 'ondisk_log_size': 3198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26145952, 'num_objects': 3192, 'num_object_clones': 0, 'num_object_copies': 12768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3196, 'num_write_kb': 25548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 875}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:56:57.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3234", 'reported_seq': 3295, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:55.901132+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:56:55.901132+0000', 'last_peered': '2024-09-11T04:56:55.901132+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:56:55.901132+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3234, 'log_dups_size': 0, 'ondisk_log_size': 3234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26449056, 'num_objects': 3229, 'num_object_clones': 0, 'num_object_copies': 12916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3232, 'num_write_kb': 25844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 868}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3275", 'reported_seq': 3330, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:56.370011+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:56:56.370011+0000', 'last_peered': '2024-09-11T04:56:56.370011+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:56:56.370011+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3275, 'log_dups_size': 0, 'ondisk_log_size': 3275, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26757440, 'num_objects': 3267, 'num_object_clones': 0, 'num_object_copies': 13068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3272, 'num_write_kb': 26153, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 873}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3103", 'reported_seq': 3178, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:53.686533+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:56:53.686533+0000', 'last_peered': '2024-09-11T04:56:53.686533+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:56:53.686533+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3103, 'log_dups_size': 0, 'ondisk_log_size': 3103, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25400480, 'num_objects': 3101, 'num_object_clones': 0, 'num_object_copies': 12404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3101, 'num_write_kb': 24806, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 783}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3172", 'reported_seq': 3227, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:56.367494+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:56:56.367494+0000', 'last_peered': '2024-09-11T04:56:56.367494+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:56:56.367494+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3172, 'log_dups_size': 0, 'ondisk_log_size': 3172, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25939696, 'num_objects': 3167, 'num_object_clones': 0, 'num_object_copies': 12668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3167, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3169, 'num_write_kb': 25340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 844}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3107", 'reported_seq': 3871, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:56.413611+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:56:56.413611+0000', 'last_peered': '2024-09-11T04:56:56.413611+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:56:56.413611+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3107, 'log_dups_size': 0, 'ondisk_log_size': 3107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25442896, 'num_objects': 3106, 'num_object_clones': 0, 'num_object_copies': 12424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1871, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3106, 'num_write_kb': 24847, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 365, 'num_bytes_recovered': 2981888, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1235}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:56:57.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3148", 'reported_seq': 3221, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:53.634385+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:56:53.634385+0000', 'last_peered': '2024-09-11T04:56:53.634385+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:56:53.634385+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3148, 'log_dups_size': 0, 'ondisk_log_size': 3148, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25778768, 'num_objects': 3147, 'num_object_clones': 0, 'num_object_copies': 12588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3147, 'num_write_kb': 25175, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:57.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:56:57.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3193", 'reported_seq': 3242, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:56.312840+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:56:56.312840+0000', 'last_peered': '2024-09-11T04:56:56.312840+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:56:56.312840+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3193, 'log_dups_size': 0, 'ondisk_log_size': 3193, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26121376, 'num_objects': 3189, 'num_object_clones': 0, 'num_object_copies': 12756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3191, 'num_write_kb': 25524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:56:57.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:56:57.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3134", 'reported_seq': 3211, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:53.699037+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:56:53.699037+0000', 'last_peered': '2024-09-11T04:56:53.699037+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:56:53.699037+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3134, 'log_dups_size': 0, 'ondisk_log_size': 3134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25646240, 'num_objects': 3131, 'num_object_clones': 0, 'num_object_copies': 12524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3132, 'num_write_kb': 25053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:56:57.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:56:57.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3197", 'reported_seq': 3258, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:55.880856+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:56:55.880856+0000', 'last_peered': '2024-09-11T04:56:55.880856+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:56:55.880856+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3197, 'log_dups_size': 0, 'ondisk_log_size': 3197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26145952, 'num_objects': 3192, 'num_object_clones': 0, 'num_object_copies': 12768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3195, 'num_write_kb': 25548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:56:57.870 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:56:58.634 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:58.634 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36518/201808 objects degraded (18.095%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":145,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50452,"data_bytes":413262016,"bytes_used":1294110720,"bytes_avail":385252945920,"bytes_total":386547056640,"degraded_objects":36518,"degraded_total":201808,"degraded_ratio":0.18095417426464755,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105106,"recovering_keys_per_sec":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"write_bytes_sec":3566116,"read_op_per_sec":0,"write_op_per_sec":435},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-11T04:56:31.620416+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:56:58.664 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:56:58.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:56:58.697 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 75 20 51546 51526 5.36619 2.64062 0.0630257 0.029103 2024-09-11T04:56:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:56:59.307 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:56:59.338 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-11T04:56:57.629368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":413262016,"num_objects":50452,"num_object_clones":0,"num_object_copies":201808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50479,"num_write_kb":403714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50506,"ondisk_log_size":50506,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1263780,"kb_used_data":658660,"kb_used_omap":14,"kb_used_meta":605041,"kb_avail":376223580,"statfs":{"total":386547056640,"available":385252945920,"internally_reserved":0,"allocated":674467840,"data_stored":673784600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":619562550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,10,16,9,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42803200,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":41800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"1.f","version":"31'3153","reported_seq":3194,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.409544+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:56:56.409544+0000","last_peered":"2024-09-11T04:56:56.409544+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:56:56.409544+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3153,"log_dups_size":0,"ondisk_log_size":3153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25829376,"num_objects":3153,"num_object_clones":0,"num_object_copies":12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":25224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3159","reported_seq":3219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.945457+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:56:55.945457+0000","last_peered":"2024-09-11T04:56:55.945457+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:56:55.945457+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25842848,"num_objects":3155,"num_object_clones":0,"num_object_copies":12620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3157,"num_write_kb":25252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2969","reported_seq":3043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.721238+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:53.721238+0000","last_peered":"2024-09-11T04:56:53.721238+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:53.721238+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":2969,"log_dups_size":0,"ondisk_log_size":2969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24296016,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":23734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3124","reported_seq":3186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.950560+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:56:55.950560+0000","last_peered":"2024-09-11T04:56:55.950560+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:56:55.950560+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3124,"log_dups_size":0,"ondisk_log_size":3124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25556128,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3122,"num_write_kb":24965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3167","reported_seq":3218,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.367948+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:56:56.367948+0000","last_peered":"2024-09-11T04:56:56.367948+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:56:56.367948+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25916576,"num_objects":3164,"num_object_clones":0,"num_object_copies":12656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3165,"num_write_kb":25317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3173","reported_seq":3213,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.302416+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:56:56.302416+0000","last_peered":"2024-09-11T04:56:56.302416+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:56:56.302416+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25993216,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3173,"num_write_kb":25384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3198","reported_seq":3260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.940178+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:56:55.940178+0000","last_peered":"2024-09-11T04:56:55.940178+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:56:55.940178+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3198,"log_dups_size":0,"ondisk_log_size":3198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26145952,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":25548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":875}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3234","reported_seq":3295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.901132+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:56:55.901132+0000","last_peered":"2024-09-11T04:56:55.901132+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:56:55.901132+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3234,"log_dups_size":0,"ondisk_log_size":3234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26449056,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3232,"num_write_kb":25844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3275","reported_seq":3330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.370011+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:56:56.370011+0000","last_peered":"2024-09-11T04:56:56.370011+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:56:56.370011+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3275,"log_dups_size":0,"ondisk_log_size":3275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26757440,"num_objects":3267,"num_object_clones":0,"num_object_copies":13068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3272,"num_write_kb":26153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3103","reported_seq":3178,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.686533+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:53.686533+0000","last_peered":"2024-09-11T04:56:53.686533+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:53.686533+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400480,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":24806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3172","reported_seq":3227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.367494+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:56:56.367494+0000","last_peered":"2024-09-11T04:56:56.367494+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:56:56.367494+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25939696,"num_objects":3167,"num_object_clones":0,"num_object_copies":12668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3169,"num_write_kb":25340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3107","reported_seq":3871,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.413611+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:56:56.413611+0000","last_peered":"2024-09-11T04:56:56.413611+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:56:56.413611+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25442896,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3106,"num_write_kb":24847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1235}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3148","reported_seq":3221,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.634385+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:53.634385+0000","last_peered":"2024-09-11T04:56:53.634385+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:53.634385+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25778768,"num_objects":3147,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3147,"num_write_kb":25175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3193","reported_seq":3242,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:56.312840+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:56:56.312840+0000","last_peered":"2024-09-11T04:56:56.312840+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:56:56.312840+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3193,"log_dups_size":0,"ondisk_log_size":3193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26121376,"num_objects":3189,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3191,"num_write_kb":25524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3134","reported_seq":3211,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:53.699037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:53.699037+0000","last_peered":"2024-09-11T04:56:53.699037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:53.699037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25646240,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3132,"num_write_kb":25053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3197","reported_seq":3258,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:55.880856+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:56:55.880856+0000","last_peered":"2024-09-11T04:56:55.880856+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:56:55.880856+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3197,"log_dups_size":0,"ondisk_log_size":3197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26145952,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":25548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":413262016,"num_objects":50452,"num_object_clones":0,"num_object_copies":201808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50479,"num_write_kb":403714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":677593088,"data_stored":677593088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50506,"ondisk_log_size":50506,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374452,"kb_used_data":202532,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93997388,"statfs":{"total":96636764160,"available":96253325312,"internally_reserved":0,"allocated":207392768,"data_stored":207221958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,8,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374392,"kb_used_data":198824,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93997448,"statfs":{"total":96636764160,"available":96253386752,"internally_reserved":0,"allocated":203595776,"data_stored":203424966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138016,"kb_used_data":56400,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94233824,"statfs":{"total":96636764160,"available":96495435776,"internally_reserved":0,"allocated":57753600,"data_stored":57582790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":83554002},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":376920,"kb_used_data":200904,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93994920,"statfs":{"total":96636764160,"available":96250798080,"internally_reserved":0,"allocated":205725696,"data_stored":205554886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207171584,"data_stored":207171584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58941440,"data_stored":58941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203563008,"data_stored":203563008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207917056,"data_stored":207917056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:56:59.697 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 76 20 51902 51882 5.33218 2.78125 0.0460572 0.0292881 2024-09-11T04:57:00.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 77 20 52284 52264 5.30169 2.98438 0.0343656 0.0294542 2024-09-11T04:57:01.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 78 20 52662 52642 5.27158 2.95312 0.0893082 0.0296272 2024-09-11T04:57:02.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:02.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 79 20 53034 53014 5.24162 2.90625 0.0322436 0.0297923 2024-09-11T04:57:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:02.973 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-11T04:57:01.630664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":428286144,"num_objects":52286,"num_object_clones":0,"num_object_copies":209144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52313,"num_write_kb":418386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52340,"ondisk_log_size":52340,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1298392,"kb_used_data":680792,"kb_used_omap":14,"kb_used_meta":617521,"kb_avail":376188968,"statfs":{"total":386547056640,"available":385217503232,"internally_reserved":0,"allocated":697131008,"data_stored":696447768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":632342070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,15,17,9,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41984000,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":41000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"1.f","version":"31'3262","reported_seq":3302,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.442050+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:01.442050+0000","last_peered":"2024-09-11T04:57:01.442050+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:01.442050+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26722304,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3269","reported_seq":3330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.897219+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:00.897219+0000","last_peered":"2024-09-11T04:57:00.897219+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:00.897219+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3269,"log_dups_size":0,"ondisk_log_size":3269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26743968,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3267,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":986}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3084","reported_seq":3158,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.692040+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:58.692040+0000","last_peered":"2024-09-11T04:56:58.692040+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:58.692040+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3084,"log_dups_size":0,"ondisk_log_size":3084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25238096,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":861}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3232","reported_seq":3294,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.991378+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:00.991378+0000","last_peered":"2024-09-11T04:57:00.991378+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:00.991378+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3232,"log_dups_size":0,"ondisk_log_size":3232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26440864,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":25829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3272","reported_seq":3325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.441046+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:01.441046+0000","last_peered":"2024-09-11T04:57:01.441046+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:01.441046+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26776736,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3270,"num_write_kb":26157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3295","reported_seq":3335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.479522+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:01.479522+0000","last_peered":"2024-09-11T04:57:01.479522+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:01.479522+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26992640,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":974}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3303","reported_seq":3365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.976293+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:00.976293+0000","last_peered":"2024-09-11T04:57:00.976293+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:00.976293+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27006112,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3301,"num_write_kb":26388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3366","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.850732+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:00.850732+0000","last_peered":"2024-09-11T04:57:00.850732+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:00.850732+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3366,"log_dups_size":0,"ondisk_log_size":3366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530400,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3364,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3394","reported_seq":3448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.481984+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:01.481984+0000","last_peered":"2024-09-11T04:57:01.481984+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:01.481984+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27732288,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3212","reported_seq":3288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.714232+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:58.714232+0000","last_peered":"2024-09-11T04:56:58.714232+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:58.714232+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3212,"log_dups_size":0,"ondisk_log_size":3212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26293408,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":25678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3289","reported_seq":3344,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.461307+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:01.461307+0000","last_peered":"2024-09-11T04:57:01.461307+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:01.461307+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26898160,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3286,"num_write_kb":26276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3217","reported_seq":4089,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.460449+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:01.460449+0000","last_peered":"2024-09-11T04:57:01.460449+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:01.460449+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26344016,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":25727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3256","reported_seq":3328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.680532+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:58.680532+0000","last_peered":"2024-09-11T04:56:58.680532+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:58.680532+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3256,"log_dups_size":0,"ondisk_log_size":3256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26663504,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3308","reported_seq":3357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.343912+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:01.343912+0000","last_peered":"2024-09-11T04:57:01.343912+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:01.343912+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27063456,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3259","reported_seq":3335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.762615+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:58.762615+0000","last_peered":"2024-09-11T04:56:58.762615+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:58.762615+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26670240,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":26053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3322","reported_seq":3383,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.991391+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:00.991391+0000","last_peered":"2024-09-11T04:57:00.991391+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:00.991391+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3322,"log_dups_size":0,"ondisk_log_size":3322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27169952,"num_objects":3317,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":428286144,"num_objects":52286,"num_object_clones":0,"num_object_copies":209144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52313,"num_write_kb":418386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":700162048,"data_stored":700162048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52340,"ondisk_log_size":52340,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385780,"kb_used_data":209700,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93986060,"statfs":{"total":96636764160,"available":96241725440,"internally_reserved":0,"allocated":214732800,"data_stored":214561990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,13,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386020,"kb_used_data":206292,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93985820,"statfs":{"total":96636764160,"available":96241479680,"internally_reserved":0,"allocated":211243008,"data_stored":211072198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138016,"kb_used_data":56400,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94233824,"statfs":{"total":96636764160,"available":96495435776,"internally_reserved":0,"allocated":57753600,"data_stored":57582790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":83554002},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388576,"kb_used_data":208400,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93983264,"statfs":{"total":96636764160,"available":96238862336,"internally_reserved":0,"allocated":213401600,"data_stored":213230790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,6,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214597632,"data_stored":214597632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58941440,"data_stored":58941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211197952,"data_stored":211197952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215425024,"data_stored":215425024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:03.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:03.616 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:03.640 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-11T04:57:01.630664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":428286144,"num_objects":52286,"num_object_clones":0,"num_object_copies":209144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52313,"num_write_kb":418386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52340,"ondisk_log_size":52340,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1298392,"kb_used_data":680792,"kb_used_omap":14,"kb_used_meta":617521,"kb_avail":376188968,"statfs":{"total":386547056640,"available":385217503232,"internally_reserved":0,"allocated":697131008,"data_stored":696447768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":632342070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,15,17,9,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41984000,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":41000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"1.f","version":"31'3262","reported_seq":3302,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.442050+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:01.442050+0000","last_peered":"2024-09-11T04:57:01.442050+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:01.442050+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26722304,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3269","reported_seq":3330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.897219+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:00.897219+0000","last_peered":"2024-09-11T04:57:00.897219+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:00.897219+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3269,"log_dups_size":0,"ondisk_log_size":3269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26743968,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3267,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":986}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3084","reported_seq":3158,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.692040+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:58.692040+0000","last_peered":"2024-09-11T04:56:58.692040+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:58.692040+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3084,"log_dups_size":0,"ondisk_log_size":3084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25238096,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":861}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3232","reported_seq":3294,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.991378+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:00.991378+0000","last_peered":"2024-09-11T04:57:00.991378+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:00.991378+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3232,"log_dups_size":0,"ondisk_log_size":3232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26440864,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":25829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3272","reported_seq":3325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.441046+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:01.441046+0000","last_peered":"2024-09-11T04:57:01.441046+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:01.441046+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26776736,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3270,"num_write_kb":26157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3295","reported_seq":3335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.479522+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:01.479522+0000","last_peered":"2024-09-11T04:57:01.479522+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:01.479522+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26992640,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":974}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3303","reported_seq":3365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.976293+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:00.976293+0000","last_peered":"2024-09-11T04:57:00.976293+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:00.976293+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27006112,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3301,"num_write_kb":26388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3366","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.850732+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:00.850732+0000","last_peered":"2024-09-11T04:57:00.850732+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:00.850732+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3366,"log_dups_size":0,"ondisk_log_size":3366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530400,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3364,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3394","reported_seq":3448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.481984+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:01.481984+0000","last_peered":"2024-09-11T04:57:01.481984+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:01.481984+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27732288,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3212","reported_seq":3288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.714232+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:58.714232+0000","last_peered":"2024-09-11T04:56:58.714232+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:58.714232+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3212,"log_dups_size":0,"ondisk_log_size":3212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26293408,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":25678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3289","reported_seq":3344,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.461307+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:01.461307+0000","last_peered":"2024-09-11T04:57:01.461307+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:01.461307+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26898160,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3286,"num_write_kb":26276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3217","reported_seq":4089,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.460449+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:01.460449+0000","last_peered":"2024-09-11T04:57:01.460449+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:01.460449+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26344016,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":25727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3256","reported_seq":3328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.680532+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:58.680532+0000","last_peered":"2024-09-11T04:56:58.680532+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:58.680532+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3256,"log_dups_size":0,"ondisk_log_size":3256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26663504,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3308","reported_seq":3357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.343912+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:01.343912+0000","last_peered":"2024-09-11T04:57:01.343912+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:01.343912+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27063456,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3259","reported_seq":3335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.762615+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:58.762615+0000","last_peered":"2024-09-11T04:56:58.762615+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:58.762615+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26670240,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":26053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3322","reported_seq":3383,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.991391+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:00.991391+0000","last_peered":"2024-09-11T04:57:00.991391+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:00.991391+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3322,"log_dups_size":0,"ondisk_log_size":3322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27169952,"num_objects":3317,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":428286144,"num_objects":52286,"num_object_clones":0,"num_object_copies":209144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52313,"num_write_kb":418386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":700162048,"data_stored":700162048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52340,"ondisk_log_size":52340,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385780,"kb_used_data":209700,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93986060,"statfs":{"total":96636764160,"available":96241725440,"internally_reserved":0,"allocated":214732800,"data_stored":214561990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,13,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386020,"kb_used_data":206292,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93985820,"statfs":{"total":96636764160,"available":96241479680,"internally_reserved":0,"allocated":211243008,"data_stored":211072198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138016,"kb_used_data":56400,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94233824,"statfs":{"total":96636764160,"available":96495435776,"internally_reserved":0,"allocated":57753600,"data_stored":57582790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":83554002},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388576,"kb_used_data":208400,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93983264,"statfs":{"total":96636764160,"available":96238862336,"internally_reserved":0,"allocated":213401600,"data_stored":213230790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,6,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214597632,"data_stored":214597632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58941440,"data_stored":58941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211197952,"data_stored":211197952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215425024,"data_stored":215425024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:03.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3262", 'reported_seq': 3302, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:01.442050+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:01.442050+0000', 'last_peered': '2024-09-11T04:57:01.442050+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:01.442050+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3262, 'log_dups_size': 0, 'ondisk_log_size': 3262, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26722304, 'num_objects': 3262, 'num_object_clones': 0, 'num_object_copies': 13048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3262, 'num_write_kb': 26096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3269", 'reported_seq': 3330, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:00.897219+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:00.897219+0000', 'last_peered': '2024-09-11T04:57:00.897219+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:00.897219+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3269, 'log_dups_size': 0, 'ondisk_log_size': 3269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26743968, 'num_objects': 3265, 'num_object_clones': 0, 'num_object_copies': 13060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3267, 'num_write_kb': 26132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 986}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3084", 'reported_seq': 3158, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:58.692040+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:56:58.692040+0000', 'last_peered': '2024-09-11T04:56:58.692040+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:56:58.692040+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3084, 'log_dups_size': 0, 'ondisk_log_size': 3084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25238096, 'num_objects': 3081, 'num_object_clones': 0, 'num_object_copies': 12324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3083, 'num_write_kb': 24654, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 861}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3232", 'reported_seq': 3294, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:00.991378+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:00.991378+0000', 'last_peered': '2024-09-11T04:57:00.991378+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:00.991378+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3232, 'log_dups_size': 0, 'ondisk_log_size': 3232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26440864, 'num_objects': 3228, 'num_object_clones': 0, 'num_object_copies': 12912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3228, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3230, 'num_write_kb': 25829, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 950}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:03.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3272", 'reported_seq': 3325, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:01.441046+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:01.441046+0000', 'last_peered': '2024-09-11T04:57:01.441046+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:01.441046+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3272, 'log_dups_size': 0, 'ondisk_log_size': 3272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26776736, 'num_objects': 3269, 'num_object_clones': 0, 'num_object_copies': 13076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3270, 'num_write_kb': 26157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1019}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3295", 'reported_seq': 3335, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:01.479522+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:01.479522+0000', 'last_peered': '2024-09-11T04:57:01.479522+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:01.479522+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3295, 'log_dups_size': 0, 'ondisk_log_size': 3295, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26992640, 'num_objects': 3295, 'num_object_clones': 0, 'num_object_copies': 13180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3295, 'num_write_kb': 26360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 974}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3303", 'reported_seq': 3365, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:00.976293+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:00.976293+0000', 'last_peered': '2024-09-11T04:57:00.976293+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:00.976293+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3303, 'log_dups_size': 0, 'ondisk_log_size': 3303, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27006112, 'num_objects': 3297, 'num_object_clones': 0, 'num_object_copies': 13188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3301, 'num_write_kb': 26388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 980}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3366", 'reported_seq': 3427, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:00.850732+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:00.850732+0000', 'last_peered': '2024-09-11T04:57:00.850732+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:00.850732+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3366, 'log_dups_size': 0, 'ondisk_log_size': 3366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27530400, 'num_objects': 3361, 'num_object_clones': 0, 'num_object_copies': 13444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3364, 'num_write_kb': 26900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1000}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3394", 'reported_seq': 3448, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:01.481984+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:01.481984+0000', 'last_peered': '2024-09-11T04:57:01.481984+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:01.481984+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3394, 'log_dups_size': 0, 'ondisk_log_size': 3394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27732288, 'num_objects': 3386, 'num_object_clones': 0, 'num_object_copies': 13544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3386, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3391, 'num_write_kb': 27105, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 992}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:03.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3212", 'reported_seq': 3288, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:58.714232+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:56:58.714232+0000', 'last_peered': '2024-09-11T04:56:58.714232+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:56:58.714232+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3212, 'log_dups_size': 0, 'ondisk_log_size': 3212, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26293408, 'num_objects': 3210, 'num_object_clones': 0, 'num_object_copies': 12840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3210, 'num_write_kb': 25678, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 892}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3289", 'reported_seq': 3344, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:01.461307+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:01.461307+0000', 'last_peered': '2024-09-11T04:57:01.461307+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:01.461307+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3289, 'log_dups_size': 0, 'ondisk_log_size': 3289, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26898160, 'num_objects': 3284, 'num_object_clones': 0, 'num_object_copies': 13136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3286, 'num_write_kb': 26276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3217", 'reported_seq': 4089, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:01.460449+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:01.460449+0000', 'last_peered': '2024-09-11T04:57:01.460449+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:01.460449+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3217, 'log_dups_size': 0, 'ondisk_log_size': 3217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26344016, 'num_objects': 3216, 'num_object_clones': 0, 'num_object_copies': 12864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1817, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3216, 'num_write_kb': 25727, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 419, 'num_bytes_recovered': 3424256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1399}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3256", 'reported_seq': 3328, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:58.680532+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:56:58.680532+0000', 'last_peered': '2024-09-11T04:56:58.680532+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:56:58.680532+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3256, 'log_dups_size': 0, 'ondisk_log_size': 3256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26663504, 'num_objects': 3255, 'num_object_clones': 0, 'num_object_copies': 13020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3255, 'num_write_kb': 26039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 923}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3308", 'reported_seq': 3357, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:01.343912+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:01.343912+0000', 'last_peered': '2024-09-11T04:57:01.343912+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:01.343912+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3308, 'log_dups_size': 0, 'ondisk_log_size': 3308, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27063456, 'num_objects': 3304, 'num_object_clones': 0, 'num_object_copies': 13216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3306, 'num_write_kb': 26444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3259", 'reported_seq': 3335, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:56:58.762615+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:56:58.762615+0000', 'last_peered': '2024-09-11T04:56:58.762615+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:56:58.762615+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3259, 'log_dups_size': 0, 'ondisk_log_size': 3259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26670240, 'num_objects': 3256, 'num_object_clones': 0, 'num_object_copies': 13024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3257, 'num_write_kb': 26053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:03.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:03.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3322", 'reported_seq': 3383, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:00.991391+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:00.991391+0000', 'last_peered': '2024-09-11T04:57:00.991391+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:00.991391+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3322, 'log_dups_size': 0, 'ondisk_log_size': 3322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27169952, 'num_objects': 3317, 'num_object_clones': 0, 'num_object_copies': 13268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3320, 'num_write_kb': 26548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1006}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:03.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:03.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:57:03.699256+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0299798 2024-09-11T04:57:03.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:57:03.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 80 20 53373 53353 5.20921 2.64844 0.102138 0.0299798 2024-09-11T04:57:04.421 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:04.421 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36464/209144 objects degraded (17.435%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":151,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52286,"data_bytes":428286144,"bytes_used":1341407232,"bytes_avail":385205649408,"bytes_total":386547056640,"degraded_objects":36464,"degraded_total":209144,"degraded_ratio":0.17434877405041502,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":67566,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"write_bytes_sec":2537482,"read_op_per_sec":0,"write_op_per_sec":309},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-11T04:56:31.620416+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:57:04.450 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:04.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:04.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 81 20 53701 53681 5.17654 2.5625 0.0370502 0.0301695 2024-09-11T04:57:05.049 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:05.072 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-11T04:57:03.631150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":428286144,"num_objects":52286,"num_object_clones":0,"num_object_copies":209144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52313,"num_write_kb":418386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52340,"ondisk_log_size":52340,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1309968,"kb_used_data":688208,"kb_used_omap":14,"kb_used_meta":621681,"kb_avail":376177392,"statfs":{"total":386547056640,"available":385205649408,"internally_reserved":0,"allocated":704724992,"data_stored":704041752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":636601910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,15,17,9,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30457856,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":29744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003179"},"pg_stats":[{"pgid":"1.f","version":"31'3262","reported_seq":3302,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.442050+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:01.442050+0000","last_peered":"2024-09-11T04:57:01.442050+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:01.442050+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26722304,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3269","reported_seq":3330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.897219+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:00.897219+0000","last_peered":"2024-09-11T04:57:00.897219+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:00.897219+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3269,"log_dups_size":0,"ondisk_log_size":3269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26743968,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3267,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":986}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3084","reported_seq":3158,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.692040+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:56:58.692040+0000","last_peered":"2024-09-11T04:56:58.692040+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:56:58.692040+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3084,"log_dups_size":0,"ondisk_log_size":3084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25238096,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":861}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3232","reported_seq":3294,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.991378+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:00.991378+0000","last_peered":"2024-09-11T04:57:00.991378+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:00.991378+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3232,"log_dups_size":0,"ondisk_log_size":3232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26440864,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":25829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3272","reported_seq":3325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.441046+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:01.441046+0000","last_peered":"2024-09-11T04:57:01.441046+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:01.441046+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26776736,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3270,"num_write_kb":26157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3295","reported_seq":3335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.479522+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:01.479522+0000","last_peered":"2024-09-11T04:57:01.479522+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:01.479522+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26992640,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":974}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3303","reported_seq":3365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.976293+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:00.976293+0000","last_peered":"2024-09-11T04:57:00.976293+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:00.976293+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27006112,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3301,"num_write_kb":26388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3366","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.850732+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:00.850732+0000","last_peered":"2024-09-11T04:57:00.850732+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:00.850732+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3366,"log_dups_size":0,"ondisk_log_size":3366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530400,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3364,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3394","reported_seq":3448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.481984+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:01.481984+0000","last_peered":"2024-09-11T04:57:01.481984+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:01.481984+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27732288,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3212","reported_seq":3288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.714232+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:56:58.714232+0000","last_peered":"2024-09-11T04:56:58.714232+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:56:58.714232+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3212,"log_dups_size":0,"ondisk_log_size":3212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26293408,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":25678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3289","reported_seq":3344,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.461307+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:01.461307+0000","last_peered":"2024-09-11T04:57:01.461307+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:01.461307+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26898160,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3286,"num_write_kb":26276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3217","reported_seq":4089,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.460449+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:01.460449+0000","last_peered":"2024-09-11T04:57:01.460449+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:01.460449+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26344016,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":25727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3256","reported_seq":3328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.680532+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:56:58.680532+0000","last_peered":"2024-09-11T04:56:58.680532+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:56:58.680532+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3256,"log_dups_size":0,"ondisk_log_size":3256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26663504,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3308","reported_seq":3357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:01.343912+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:01.343912+0000","last_peered":"2024-09-11T04:57:01.343912+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:01.343912+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27063456,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3259","reported_seq":3335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:56:58.762615+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:56:58.762615+0000","last_peered":"2024-09-11T04:56:58.762615+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:56:58.762615+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26670240,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":26053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3322","reported_seq":3383,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:00.991391+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:00.991391+0000","last_peered":"2024-09-11T04:57:00.991391+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:00.991391+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3322,"log_dups_size":0,"ondisk_log_size":3322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27169952,"num_objects":3317,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":428286144,"num_objects":52286,"num_object_clones":0,"num_object_copies":209144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52313,"num_write_kb":418386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":3424256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":707862528,"data_stored":707862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52340,"ondisk_log_size":52340,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385780,"kb_used_data":209700,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93986060,"statfs":{"total":96636764160,"available":96241725440,"internally_reserved":0,"allocated":214732800,"data_stored":214561990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,13,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386020,"kb_used_data":206292,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93985820,"statfs":{"total":96636764160,"available":96241479680,"internally_reserved":0,"allocated":211243008,"data_stored":211072198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149592,"kb_used_data":63816,"kb_used_omap":4,"kb_used_meta":85755,"kb_avail":94222248,"statfs":{"total":96636764160,"available":96483581952,"internally_reserved":0,"allocated":65347584,"data_stored":65176774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":87813842},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388576,"kb_used_data":208400,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93983264,"statfs":{"total":96636764160,"available":96238862336,"internally_reserved":0,"allocated":213401600,"data_stored":213230790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,6,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214597632,"data_stored":214597632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66641920,"data_stored":66641920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211197952,"data_stored":211197952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215425024,"data_stored":215425024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:05.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 82 20 54040 54020 5.14571 2.64844 0.0460049 0.0303497 2024-09-11T04:57:06.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 83 20 54426 54406 5.12004 3.01562 0.0886363 0.0305025 2024-09-11T04:57:07.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 84 20 54791 54771 5.09304 2.85156 0.0453689 0.0306628 2024-09-11T04:57:08.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:08.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 85 20 55134 55114 5.06465 2.67969 0.0720976 0.0308309 2024-09-11T04:57:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:08.746 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-11T04:57:07.632062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":443039936,"num_objects":54087,"num_object_clones":0,"num_object_copies":216348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54114,"num_write_kb":432794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54141,"ondisk_log_size":54141,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1359272,"kb_used_data":716712,"kb_used_omap":14,"kb_used_meta":642481,"kb_avail":376128088,"statfs":{"total":386547056640,"available":385155162112,"internally_reserved":0,"allocated":733913088,"data_stored":733229848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":657901110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,7,17,15,11,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41451520,"num_objects":5060,"num_object_clones":0,"num_object_copies":20240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":40480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.f","version":"31'3348","reported_seq":3389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.491818+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:06.491818+0000","last_peered":"2024-09-11T04:57:06.491818+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:06.491818+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27426816,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3387","reported_seq":3448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.042120+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:06.042120+0000","last_peered":"2024-09-11T04:57:06.042120+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:06.042120+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3387,"log_dups_size":0,"ondisk_log_size":3387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27710624,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":27076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3192","reported_seq":3265,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:03.693523+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:03.693523+0000","last_peered":"2024-09-11T04:57:03.693523+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:03.693523+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3192,"log_dups_size":0,"ondisk_log_size":3192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26122832,"num_objects":3189,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3191,"num_write_kb":25518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3350","reported_seq":3412,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:05.956614+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:05.956614+0000","last_peered":"2024-09-11T04:57:05.956614+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:05.956614+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27407520,"num_objects":3346,"num_object_clones":0,"num_object_copies":13384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3385","reported_seq":3438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.475650+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:06.475650+0000","last_peered":"2024-09-11T04:57:06.475650+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:06.475650+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3385,"log_dups_size":0,"ondisk_log_size":3385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27702432,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3383,"num_write_kb":27061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3421","reported_seq":3461,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.514804+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:06.514804+0000","last_peered":"2024-09-11T04:57:06.514804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:06.514804+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3421,"log_dups_size":0,"ondisk_log_size":3421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28024832,"num_objects":3421,"num_object_clones":0,"num_object_copies":13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3419","reported_seq":3481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.015526+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:06.015526+0000","last_peered":"2024-09-11T04:57:06.015526+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:06.015526+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3419,"log_dups_size":0,"ondisk_log_size":3419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956384,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3469","reported_seq":3530,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.001097+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:06.001097+0000","last_peered":"2024-09-11T04:57:06.001097+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:06.001097+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28374176,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3467,"num_write_kb":27724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3495","reported_seq":3550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.463590+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:06.463590+0000","last_peered":"2024-09-11T04:57:06.463590+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:06.463590+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28559680,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3492,"num_write_kb":27913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1093}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3332","reported_seq":3408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:03.785480+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:03.785480+0000","last_peered":"2024-09-11T04:57:03.785480+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:03.785480+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276448,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3330,"num_write_kb":26638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3394","reported_seq":3449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.456163+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:06.456163+0000","last_peered":"2024-09-11T04:57:06.456163+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:06.456163+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27758320,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3347","reported_seq":4327,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.582370+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:06.582370+0000","last_peered":"2024-09-11T04:57:06.582370+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:06.582370+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3347,"log_dups_size":0,"ondisk_log_size":3347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27408976,"num_objects":3346,"num_object_clones":0,"num_object_copies":13384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":26767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1583}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3384","reported_seq":3457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:03.702508+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:03.702508+0000","last_peered":"2024-09-11T04:57:03.702508+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:03.702508+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27712080,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3383,"num_write_kb":27063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1051}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3409","reported_seq":3458,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.381778+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:06.381778+0000","last_peered":"2024-09-11T04:57:06.381778+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:06.381778+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3409,"log_dups_size":0,"ondisk_log_size":3409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27890848,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3407,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3375","reported_seq":3452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:03.702876+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:03.702876+0000","last_peered":"2024-09-11T04:57:03.702876+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:03.702876+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3375,"log_dups_size":0,"ondisk_log_size":3375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620512,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3373,"num_write_kb":26981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1054}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3434","reported_seq":3496,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:05.997677+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:05.997677+0000","last_peered":"2024-09-11T04:57:05.997677+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:05.997677+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":27444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":443039936,"num_objects":54087,"num_object_clones":0,"num_object_copies":216348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54114,"num_write_kb":432794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737439744,"data_stored":737439744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54141,"ondisk_log_size":54141,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396792,"kb_used_data":216552,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93975048,"statfs":{"total":96636764160,"available":96230449152,"internally_reserved":0,"allocated":221749248,"data_stored":221578438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,9,2,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397348,"kb_used_data":213460,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93974492,"statfs":{"total":96636764160,"available":96229879808,"internally_reserved":0,"allocated":218583040,"data_stored":218412230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165364,"kb_used_data":71268,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94206476,"statfs":{"total":96636764160,"available":96467431424,"internally_reserved":0,"allocated":72978432,"data_stored":72807622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96333522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399768,"kb_used_data":215432,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93972072,"statfs":{"total":96636764160,"available":96227401728,"internally_reserved":0,"allocated":220602368,"data_stored":220431558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188739279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,4,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221900800,"data_stored":221900800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74088448,"data_stored":74088448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218730496,"data_stored":218730496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":222720000,"data_stored":222720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:08.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:09.344 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:09.365 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-11T04:57:07.632062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":443039936,"num_objects":54087,"num_object_clones":0,"num_object_copies":216348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54114,"num_write_kb":432794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54141,"ondisk_log_size":54141,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1359272,"kb_used_data":716712,"kb_used_omap":14,"kb_used_meta":642481,"kb_avail":376128088,"statfs":{"total":386547056640,"available":385155162112,"internally_reserved":0,"allocated":733913088,"data_stored":733229848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":657901110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,7,17,15,11,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41451520,"num_objects":5060,"num_object_clones":0,"num_object_copies":20240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":40480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.f","version":"31'3348","reported_seq":3389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.491818+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:06.491818+0000","last_peered":"2024-09-11T04:57:06.491818+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:06.491818+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27426816,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3387","reported_seq":3448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.042120+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:06.042120+0000","last_peered":"2024-09-11T04:57:06.042120+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:06.042120+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3387,"log_dups_size":0,"ondisk_log_size":3387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27710624,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":27076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3192","reported_seq":3265,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:03.693523+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:03.693523+0000","last_peered":"2024-09-11T04:57:03.693523+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:03.693523+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3192,"log_dups_size":0,"ondisk_log_size":3192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26122832,"num_objects":3189,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3191,"num_write_kb":25518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3350","reported_seq":3412,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:05.956614+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:05.956614+0000","last_peered":"2024-09-11T04:57:05.956614+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:05.956614+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27407520,"num_objects":3346,"num_object_clones":0,"num_object_copies":13384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3385","reported_seq":3438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.475650+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:06.475650+0000","last_peered":"2024-09-11T04:57:06.475650+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:06.475650+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3385,"log_dups_size":0,"ondisk_log_size":3385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27702432,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3383,"num_write_kb":27061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3421","reported_seq":3461,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.514804+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:06.514804+0000","last_peered":"2024-09-11T04:57:06.514804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:06.514804+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3421,"log_dups_size":0,"ondisk_log_size":3421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28024832,"num_objects":3421,"num_object_clones":0,"num_object_copies":13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3419","reported_seq":3481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.015526+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:06.015526+0000","last_peered":"2024-09-11T04:57:06.015526+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:06.015526+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3419,"log_dups_size":0,"ondisk_log_size":3419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956384,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3469","reported_seq":3530,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.001097+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:06.001097+0000","last_peered":"2024-09-11T04:57:06.001097+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:06.001097+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28374176,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3467,"num_write_kb":27724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3495","reported_seq":3550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.463590+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:06.463590+0000","last_peered":"2024-09-11T04:57:06.463590+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:06.463590+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28559680,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3492,"num_write_kb":27913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1093}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3332","reported_seq":3408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:03.785480+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:03.785480+0000","last_peered":"2024-09-11T04:57:03.785480+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:03.785480+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276448,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3330,"num_write_kb":26638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3394","reported_seq":3449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.456163+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:06.456163+0000","last_peered":"2024-09-11T04:57:06.456163+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:06.456163+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27758320,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3347","reported_seq":4327,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.582370+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:06.582370+0000","last_peered":"2024-09-11T04:57:06.582370+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:06.582370+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3347,"log_dups_size":0,"ondisk_log_size":3347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27408976,"num_objects":3346,"num_object_clones":0,"num_object_copies":13384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":26767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1583}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3384","reported_seq":3457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:03.702508+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:03.702508+0000","last_peered":"2024-09-11T04:57:03.702508+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:03.702508+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27712080,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3383,"num_write_kb":27063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1051}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3409","reported_seq":3458,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.381778+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:06.381778+0000","last_peered":"2024-09-11T04:57:06.381778+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:06.381778+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3409,"log_dups_size":0,"ondisk_log_size":3409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27890848,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3407,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3375","reported_seq":3452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:03.702876+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:03.702876+0000","last_peered":"2024-09-11T04:57:03.702876+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:03.702876+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3375,"log_dups_size":0,"ondisk_log_size":3375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620512,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3373,"num_write_kb":26981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1054}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3434","reported_seq":3496,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:05.997677+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:05.997677+0000","last_peered":"2024-09-11T04:57:05.997677+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:05.997677+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":27444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":443039936,"num_objects":54087,"num_object_clones":0,"num_object_copies":216348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54114,"num_write_kb":432794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737439744,"data_stored":737439744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54141,"ondisk_log_size":54141,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396792,"kb_used_data":216552,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93975048,"statfs":{"total":96636764160,"available":96230449152,"internally_reserved":0,"allocated":221749248,"data_stored":221578438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,9,2,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397348,"kb_used_data":213460,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93974492,"statfs":{"total":96636764160,"available":96229879808,"internally_reserved":0,"allocated":218583040,"data_stored":218412230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165364,"kb_used_data":71268,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94206476,"statfs":{"total":96636764160,"available":96467431424,"internally_reserved":0,"allocated":72978432,"data_stored":72807622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96333522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399768,"kb_used_data":215432,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93972072,"statfs":{"total":96636764160,"available":96227401728,"internally_reserved":0,"allocated":220602368,"data_stored":220431558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188739279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,4,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221900800,"data_stored":221900800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74088448,"data_stored":74088448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218730496,"data_stored":218730496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":222720000,"data_stored":222720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:09.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:09.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3348", 'reported_seq': 3389, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.491818+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:06.491818+0000', 'last_peered': '2024-09-11T04:57:06.491818+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:06.491818+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3348, 'log_dups_size': 0, 'ondisk_log_size': 3348, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27426816, 'num_objects': 3348, 'num_object_clones': 0, 'num_object_copies': 13392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3348, 'num_write_kb': 26784, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1036}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:09.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:09.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3387", 'reported_seq': 3448, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.042120+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:06.042120+0000', 'last_peered': '2024-09-11T04:57:06.042120+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:06.042120+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3387, 'log_dups_size': 0, 'ondisk_log_size': 3387, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27710624, 'num_objects': 3383, 'num_object_clones': 0, 'num_object_copies': 13532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3385, 'num_write_kb': 27076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:09.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:09.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3192", 'reported_seq': 3265, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:03.693523+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:03.693523+0000', 'last_peered': '2024-09-11T04:57:03.693523+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:03.693523+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3192, 'log_dups_size': 0, 'ondisk_log_size': 3192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26122832, 'num_objects': 3189, 'num_object_clones': 0, 'num_object_copies': 12756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3191, 'num_write_kb': 25518, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 969}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3350", 'reported_seq': 3412, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:05.956614+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:05.956614+0000', 'last_peered': '2024-09-11T04:57:05.956614+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:05.956614+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3350, 'log_dups_size': 0, 'ondisk_log_size': 3350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27407520, 'num_objects': 3346, 'num_object_clones': 0, 'num_object_copies': 13384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3348, 'num_write_kb': 26773, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1068}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3385", 'reported_seq': 3438, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.475650+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:06.475650+0000', 'last_peered': '2024-09-11T04:57:06.475650+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:06.475650+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3385, 'log_dups_size': 0, 'ondisk_log_size': 3385, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27702432, 'num_objects': 3382, 'num_object_clones': 0, 'num_object_copies': 13528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3382, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3383, 'num_write_kb': 27061, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3421", 'reported_seq': 3461, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.514804+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:06.514804+0000', 'last_peered': '2024-09-11T04:57:06.514804+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:06.514804+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3421, 'log_dups_size': 0, 'ondisk_log_size': 3421, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28024832, 'num_objects': 3421, 'num_object_clones': 0, 'num_object_copies': 13684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3421, 'num_write_kb': 27368, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3419", 'reported_seq': 3481, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.015526+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:06.015526+0000', 'last_peered': '2024-09-11T04:57:06.015526+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:06.015526+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3419, 'log_dups_size': 0, 'ondisk_log_size': 3419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27956384, 'num_objects': 3413, 'num_object_clones': 0, 'num_object_copies': 13652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3417, 'num_write_kb': 27316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1096}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:09.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3469", 'reported_seq': 3530, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.001097+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:06.001097+0000', 'last_peered': '2024-09-11T04:57:06.001097+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:06.001097+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3469, 'log_dups_size': 0, 'ondisk_log_size': 3469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28374176, 'num_objects': 3464, 'num_object_clones': 0, 'num_object_copies': 13856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3467, 'num_write_kb': 27724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3495", 'reported_seq': 3550, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.463590+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:06.463590+0000', 'last_peered': '2024-09-11T04:57:06.463590+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:06.463590+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3495, 'log_dups_size': 0, 'ondisk_log_size': 3495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28559680, 'num_objects': 3487, 'num_object_clones': 0, 'num_object_copies': 13948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3492, 'num_write_kb': 27913, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1093}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3332", 'reported_seq': 3408, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:03.785480+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:03.785480+0000', 'last_peered': '2024-09-11T04:57:03.785480+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:03.785480+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3332, 'log_dups_size': 0, 'ondisk_log_size': 3332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27276448, 'num_objects': 3330, 'num_object_clones': 0, 'num_object_copies': 13320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3330, 'num_write_kb': 26638, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1012}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3394", 'reported_seq': 3449, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.456163+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:06.456163+0000', 'last_peered': '2024-09-11T04:57:06.456163+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:06.456163+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3394, 'log_dups_size': 0, 'ondisk_log_size': 3394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27758320, 'num_objects': 3389, 'num_object_clones': 0, 'num_object_copies': 13556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3391, 'num_write_kb': 27116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1066}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3347", 'reported_seq': 4327, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.582370+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:06.582370+0000', 'last_peered': '2024-09-11T04:57:06.582370+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:06.582370+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3347, 'log_dups_size': 0, 'ondisk_log_size': 3347, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27408976, 'num_objects': 3346, 'num_object_clones': 0, 'num_object_copies': 13384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1763, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3346, 'num_write_kb': 26767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 473, 'num_bytes_recovered': 3866624, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1583}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:09.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3384", 'reported_seq': 3457, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:03.702508+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:03.702508+0000', 'last_peered': '2024-09-11T04:57:03.702508+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:03.702508+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3384, 'log_dups_size': 0, 'ondisk_log_size': 3384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27712080, 'num_objects': 3383, 'num_object_clones': 0, 'num_object_copies': 13532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3383, 'num_write_kb': 27063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1051}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:09.369 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:09.369 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3409", 'reported_seq': 3458, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:06.381778+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:06.381778+0000', 'last_peered': '2024-09-11T04:57:06.381778+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:06.381778+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3409, 'log_dups_size': 0, 'ondisk_log_size': 3409, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27890848, 'num_objects': 3405, 'num_object_clones': 0, 'num_object_copies': 13620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3407, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:09.369 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:09.369 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3375", 'reported_seq': 3452, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:03.702876+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:03.702876+0000', 'last_peered': '2024-09-11T04:57:03.702876+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:03.702876+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3375, 'log_dups_size': 0, 'ondisk_log_size': 3375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27620512, 'num_objects': 3372, 'num_object_clones': 0, 'num_object_copies': 13488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3373, 'num_write_kb': 26981, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:09.369 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:09.369 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3434", 'reported_seq': 3496, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:05.997677+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:05.997677+0000', 'last_peered': '2024-09-11T04:57:05.997677+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:05.997677+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3434, 'log_dups_size': 0, 'ondisk_log_size': 3434, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28087456, 'num_objects': 3429, 'num_object_clones': 0, 'num_object_copies': 13716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3432, 'num_write_kb': 27444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:09.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:09.698 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 86 20 55463 55443 5.03564 2.57031 0.0624898 0.0310112 2024-09-11T04:57:10.133 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:10.134 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36410/218220 objects degraded (16.685%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":157,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54555,"data_bytes":446873792,"bytes_used":1386315776,"bytes_avail":385160740864,"bytes_total":386547056640,"degraded_objects":36410,"degraded_total":218220,"degraded_ratio":0.16684996792228027,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":73708,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"write_bytes_sec":2800236,"read_op_per_sec":0,"write_op_per_sec":341},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-11T04:56:31.620416+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:57:10.157 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:10.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:10.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 87 20 55779 55759 5.00614 2.46875 0.0858722 0.0311936 2024-09-11T04:57:10.802 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:10.803 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:10.830 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-11T04:57:09.632560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446873792,"num_objects":54555,"num_object_clones":0,"num_object_copies":218220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54582,"num_write_kb":436538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54609,"ondisk_log_size":54609,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1353824,"kb_used_data":723744,"kb_used_omap":14,"kb_used_meta":630001,"kb_avail":376133536,"statfs":{"total":386547056640,"available":385160740864,"internally_reserved":0,"allocated":741113856,"data_stored":740430616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":645121590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,13,8,15,16,11,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33611776,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4103,"num_write_kb":32824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003192"},"pg_stats":[{"pgid":"1.f","version":"31'3348","reported_seq":3389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.491818+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:06.491818+0000","last_peered":"2024-09-11T04:57:06.491818+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:06.491818+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27426816,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3387","reported_seq":3448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.042120+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:06.042120+0000","last_peered":"2024-09-11T04:57:06.042120+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:06.042120+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3387,"log_dups_size":0,"ondisk_log_size":3387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27710624,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":27076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3310","reported_seq":3384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.720162+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:08.720162+0000","last_peered":"2024-09-11T04:57:08.720162+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:08.720162+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27089488,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3309,"num_write_kb":26462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3350","reported_seq":3412,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:05.956614+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:05.956614+0000","last_peered":"2024-09-11T04:57:05.956614+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:05.956614+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27407520,"num_objects":3346,"num_object_clones":0,"num_object_copies":13384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3385","reported_seq":3438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.475650+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:06.475650+0000","last_peered":"2024-09-11T04:57:06.475650+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:06.475650+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3385,"log_dups_size":0,"ondisk_log_size":3385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27702432,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3383,"num_write_kb":27061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3421","reported_seq":3461,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.514804+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:06.514804+0000","last_peered":"2024-09-11T04:57:06.514804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:06.514804+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3421,"log_dups_size":0,"ondisk_log_size":3421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28024832,"num_objects":3421,"num_object_clones":0,"num_object_copies":13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3419","reported_seq":3481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.015526+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:06.015526+0000","last_peered":"2024-09-11T04:57:06.015526+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:06.015526+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3419,"log_dups_size":0,"ondisk_log_size":3419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956384,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":27316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1096}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3469","reported_seq":3530,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.001097+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:06.001097+0000","last_peered":"2024-09-11T04:57:06.001097+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:06.001097+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28374176,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3467,"num_write_kb":27724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3495","reported_seq":3550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.463590+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:06.463590+0000","last_peered":"2024-09-11T04:57:06.463590+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:06.463590+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28559680,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3492,"num_write_kb":27913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1093}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3450","reported_seq":3525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.797309+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:08.797309+0000","last_peered":"2024-09-11T04:57:08.797309+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:08.797309+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28243104,"num_objects":3448,"num_object_clones":0,"num_object_copies":13792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1130}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3394","reported_seq":3449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.456163+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:06.456163+0000","last_peered":"2024-09-11T04:57:06.456163+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:06.456163+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27758320,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3347","reported_seq":4327,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.582370+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:06.582370+0000","last_peered":"2024-09-11T04:57:06.582370+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:06.582370+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3347,"log_dups_size":0,"ondisk_log_size":3347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27408976,"num_objects":3346,"num_object_clones":0,"num_object_copies":13384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":26767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1583}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3514","reported_seq":3587,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.785819+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:08.785819+0000","last_peered":"2024-09-11T04:57:08.785819+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:08.785819+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28777040,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3409","reported_seq":3458,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:06.381778+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:06.381778+0000","last_peered":"2024-09-11T04:57:06.381778+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:06.381778+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3409,"log_dups_size":0,"ondisk_log_size":3409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27890848,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3407,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3477","reported_seq":3554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.748273+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:08.748273+0000","last_peered":"2024-09-11T04:57:08.748273+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:08.748273+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3477,"log_dups_size":0,"ondisk_log_size":3477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3475,"num_write_kb":27797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1156}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3434","reported_seq":3496,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:05.997677+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:05.997677+0000","last_peered":"2024-09-11T04:57:05.997677+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:05.997677+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":27444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":446873792,"num_objects":54555,"num_object_clones":0,"num_object_copies":218220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54582,"num_write_kb":436538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":744714240,"data_stored":744714240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54609,"ondisk_log_size":54609,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396792,"kb_used_data":216552,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93975048,"statfs":{"total":96636764160,"available":96230449152,"internally_reserved":0,"allocated":221749248,"data_stored":221578438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,9,2,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391900,"kb_used_data":220492,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93979940,"statfs":{"total":96636764160,"available":96235458560,"internally_reserved":0,"allocated":225783808,"data_stored":225612998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,2,0,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165364,"kb_used_data":71268,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94206476,"statfs":{"total":96636764160,"available":96467431424,"internally_reserved":0,"allocated":72978432,"data_stored":72807622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":96333522},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399768,"kb_used_data":215432,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93972072,"statfs":{"total":96636764160,"available":96227401728,"internally_reserved":0,"allocated":220602368,"data_stored":220431558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188739279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,4,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221900800,"data_stored":221900800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74088448,"data_stored":74088448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":226004992,"data_stored":226004992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":222720000,"data_stored":222720000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:11.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 88 20 56149 56129 4.9821 2.89062 0.0453526 0.031348 2024-09-11T04:57:12.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 89 20 56523 56503 4.95895 2.92188 0.0306295 0.0314951 2024-09-11T04:57:13.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 90 20 56886 56866 4.93536 2.83594 0.068391 0.031642 2024-09-11T04:57:13.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:14.488 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:14.488 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:14.517 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-11T04:57:13.634147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":457425088,"num_objects":55843,"num_object_clones":0,"num_object_copies":223372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55870,"num_write_kb":446842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55897,"ondisk_log_size":55897,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1380852,"kb_used_data":745204,"kb_used_omap":14,"kb_used_meta":635569,"kb_avail":376106508,"statfs":{"total":386547056640,"available":385133064192,"internally_reserved":0,"allocated":763088896,"data_stored":762405656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":650823222},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,12,5,4,17,17,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29138944,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":827392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003482"},"pg_stats":[{"pgid":"1.f","version":"31'3455","reported_seq":3496,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.569276+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:11.569276+0000","last_peered":"2024-09-11T04:57:11.569276+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:11.569276+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3455,"log_dups_size":0,"ondisk_log_size":3455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28303360,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3501","reported_seq":3562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:10.932773+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:10.932773+0000","last_peered":"2024-09-11T04:57:10.932773+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:10.932773+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28644512,"num_objects":3497,"num_object_clones":0,"num_object_copies":13988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":27988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3310","reported_seq":3384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.720162+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:08.720162+0000","last_peered":"2024-09-11T04:57:08.720162+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:08.720162+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27089488,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3309,"num_write_kb":26462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3450","reported_seq":3511,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.035183+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:11.035183+0000","last_peered":"2024-09-11T04:57:11.035183+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:11.035183+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28226720,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3495","reported_seq":3548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.595784+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:11.595784+0000","last_peered":"2024-09-11T04:57:11.595784+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:11.595784+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603552,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":27941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3522","reported_seq":3562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.477319+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:11.477319+0000","last_peered":"2024-09-11T04:57:11.477319+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:11.477319+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3522,"log_dups_size":0,"ondisk_log_size":3522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28852224,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":28176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3530","reported_seq":3591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.023897+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:11.023897+0000","last_peered":"2024-09-11T04:57:11.023897+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:11.023897+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28865696,"num_objects":3524,"num_object_clones":0,"num_object_copies":14096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1207}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3569","reported_seq":3629,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.041728+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:11.041728+0000","last_peered":"2024-09-11T04:57:11.041728+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:11.041728+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29193376,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3567,"num_write_kb":28524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3607","reported_seq":3659,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.507999+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:11.507999+0000","last_peered":"2024-09-11T04:57:11.507999+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:11.507999+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3607,"log_dups_size":0,"ondisk_log_size":3607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29477184,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3450","reported_seq":3525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.797309+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:08.797309+0000","last_peered":"2024-09-11T04:57:08.797309+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:08.797309+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28243104,"num_objects":3448,"num_object_clones":0,"num_object_copies":13792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1130}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3504","reported_seq":3558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.610825+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:11.610825+0000","last_peered":"2024-09-11T04:57:11.610825+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:11.610825+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28659440,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":27996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3459","reported_seq":4532,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.555733+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:11.555733+0000","last_peered":"2024-09-11T04:57:11.555733+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:11.555733+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3459,"log_dups_size":0,"ondisk_log_size":3459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28326480,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3458,"num_write_kb":27663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1742}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3514","reported_seq":3587,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.785819+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:08.785819+0000","last_peered":"2024-09-11T04:57:08.785819+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:08.785819+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28777040,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3521","reported_seq":3570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.507232+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:11.507232+0000","last_peered":"2024-09-11T04:57:11.507232+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:11.507232+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28808352,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":28148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1230}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3477","reported_seq":3554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.748273+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:08.748273+0000","last_peered":"2024-09-11T04:57:08.748273+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:08.748273+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3477,"log_dups_size":0,"ondisk_log_size":3477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3475,"num_write_kb":27797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1156}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3533","reported_seq":3595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.008583+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:11.008583+0000","last_peered":"2024-09-11T04:57:11.008583+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:11.008583+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28898464,"num_objects":3528,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":457425088,"num_objects":55843,"num_object_clones":0,"num_object_copies":223372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55870,"num_write_kb":446842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":766349312,"data_stored":766349312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55897,"ondisk_log_size":55897,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407996,"kb_used_data":223596,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93963844,"statfs":{"total":96636764160,"available":96218976256,"internally_reserved":0,"allocated":228962304,"data_stored":228791494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,7,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391900,"kb_used_data":220492,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93979940,"statfs":{"total":96636764160,"available":96235458560,"internally_reserved":0,"allocated":225783808,"data_stored":225612998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,2,0,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182236,"kb_used_data":78412,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94189604,"statfs":{"total":96636764160,"available":96450154496,"internally_reserved":0,"allocated":80293888,"data_stored":80123078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":106294994},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398720,"kb_used_data":222704,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93973120,"statfs":{"total":96636764160,"available":96228474880,"internally_reserved":0,"allocated":228048896,"data_stored":227878086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229019648,"data_stored":229019648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81457152,"data_stored":81457152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":226004992,"data_stored":226004992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229867520,"data_stored":229867520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:14.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:14.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 91 20 57221 57201 4.90988 2.61719 0.0777447 0.0318065 2024-09-11T04:57:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:15.143 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:15.170 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-11T04:57:13.634147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":457425088,"num_objects":55843,"num_object_clones":0,"num_object_copies":223372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55870,"num_write_kb":446842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55897,"ondisk_log_size":55897,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1380852,"kb_used_data":745204,"kb_used_omap":14,"kb_used_meta":635569,"kb_avail":376106508,"statfs":{"total":386547056640,"available":385133064192,"internally_reserved":0,"allocated":763088896,"data_stored":762405656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":650823222},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,12,5,4,17,17,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29138944,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":827392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003482"},"pg_stats":[{"pgid":"1.f","version":"31'3455","reported_seq":3496,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.569276+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:11.569276+0000","last_peered":"2024-09-11T04:57:11.569276+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:11.569276+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3455,"log_dups_size":0,"ondisk_log_size":3455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28303360,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3501","reported_seq":3562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:10.932773+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:10.932773+0000","last_peered":"2024-09-11T04:57:10.932773+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:10.932773+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28644512,"num_objects":3497,"num_object_clones":0,"num_object_copies":13988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":27988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3310","reported_seq":3384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.720162+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:08.720162+0000","last_peered":"2024-09-11T04:57:08.720162+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:08.720162+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27089488,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3309,"num_write_kb":26462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3450","reported_seq":3511,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.035183+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:11.035183+0000","last_peered":"2024-09-11T04:57:11.035183+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:11.035183+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28226720,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3495","reported_seq":3548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.595784+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:11.595784+0000","last_peered":"2024-09-11T04:57:11.595784+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:11.595784+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603552,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":27941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3522","reported_seq":3562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.477319+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:11.477319+0000","last_peered":"2024-09-11T04:57:11.477319+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:11.477319+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3522,"log_dups_size":0,"ondisk_log_size":3522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28852224,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":28176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3530","reported_seq":3591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.023897+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:11.023897+0000","last_peered":"2024-09-11T04:57:11.023897+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:11.023897+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28865696,"num_objects":3524,"num_object_clones":0,"num_object_copies":14096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1207}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3569","reported_seq":3629,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.041728+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:11.041728+0000","last_peered":"2024-09-11T04:57:11.041728+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:11.041728+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29193376,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3567,"num_write_kb":28524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3607","reported_seq":3659,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.507999+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:11.507999+0000","last_peered":"2024-09-11T04:57:11.507999+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:11.507999+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3607,"log_dups_size":0,"ondisk_log_size":3607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29477184,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3450","reported_seq":3525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.797309+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:08.797309+0000","last_peered":"2024-09-11T04:57:08.797309+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:08.797309+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28243104,"num_objects":3448,"num_object_clones":0,"num_object_copies":13792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1130}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3504","reported_seq":3558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.610825+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:11.610825+0000","last_peered":"2024-09-11T04:57:11.610825+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:11.610825+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28659440,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":27996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3459","reported_seq":4532,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.555733+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:11.555733+0000","last_peered":"2024-09-11T04:57:11.555733+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:11.555733+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3459,"log_dups_size":0,"ondisk_log_size":3459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28326480,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3458,"num_write_kb":27663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1742}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3514","reported_seq":3587,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.785819+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:08.785819+0000","last_peered":"2024-09-11T04:57:08.785819+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:08.785819+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28777040,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3521","reported_seq":3570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.507232+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:11.507232+0000","last_peered":"2024-09-11T04:57:11.507232+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:11.507232+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28808352,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":28148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1230}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3477","reported_seq":3554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:08.748273+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:08.748273+0000","last_peered":"2024-09-11T04:57:08.748273+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:08.748273+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3477,"log_dups_size":0,"ondisk_log_size":3477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456096,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3475,"num_write_kb":27797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1156}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3533","reported_seq":3595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.008583+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:11.008583+0000","last_peered":"2024-09-11T04:57:11.008583+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:11.008583+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28898464,"num_objects":3528,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":457425088,"num_objects":55843,"num_object_clones":0,"num_object_copies":223372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55870,"num_write_kb":446842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":766349312,"data_stored":766349312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55897,"ondisk_log_size":55897,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407996,"kb_used_data":223596,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93963844,"statfs":{"total":96636764160,"available":96218976256,"internally_reserved":0,"allocated":228962304,"data_stored":228791494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,7,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391900,"kb_used_data":220492,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93979940,"statfs":{"total":96636764160,"available":96235458560,"internally_reserved":0,"allocated":225783808,"data_stored":225612998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,2,0,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182236,"kb_used_data":78412,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94189604,"statfs":{"total":96636764160,"available":96450154496,"internally_reserved":0,"allocated":80293888,"data_stored":80123078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":106294994},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398720,"kb_used_data":222704,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93973120,"statfs":{"total":96636764160,"available":96228474880,"internally_reserved":0,"allocated":228048896,"data_stored":227878086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229019648,"data_stored":229019648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81457152,"data_stored":81457152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":226004992,"data_stored":226004992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229867520,"data_stored":229867520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3455", 'reported_seq': 3496, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.569276+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:11.569276+0000', 'last_peered': '2024-09-11T04:57:11.569276+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:11.569276+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3455, 'log_dups_size': 0, 'ondisk_log_size': 3455, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28303360, 'num_objects': 3455, 'num_object_clones': 0, 'num_object_copies': 13820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3455, 'num_write_kb': 27640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3501", 'reported_seq': 3562, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:10.932773+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:10.932773+0000', 'last_peered': '2024-09-11T04:57:10.932773+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:10.932773+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3501, 'log_dups_size': 0, 'ondisk_log_size': 3501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28644512, 'num_objects': 3497, 'num_object_clones': 0, 'num_object_copies': 13988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3499, 'num_write_kb': 27988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3310", 'reported_seq': 3384, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:08.720162+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:08.720162+0000', 'last_peered': '2024-09-11T04:57:08.720162+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:08.720162+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3310, 'log_dups_size': 0, 'ondisk_log_size': 3310, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27089488, 'num_objects': 3307, 'num_object_clones': 0, 'num_object_copies': 13228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3309, 'num_write_kb': 26462, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1087}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3450", 'reported_seq': 3511, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.035183+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:11.035183+0000', 'last_peered': '2024-09-11T04:57:11.035183+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:11.035183+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3450, 'log_dups_size': 0, 'ondisk_log_size': 3450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28226720, 'num_objects': 3446, 'num_object_clones': 0, 'num_object_copies': 13784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3448, 'num_write_kb': 27573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:15.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3495", 'reported_seq': 3548, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.595784+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:11.595784+0000', 'last_peered': '2024-09-11T04:57:11.595784+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:11.595784+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3495, 'log_dups_size': 0, 'ondisk_log_size': 3495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28603552, 'num_objects': 3492, 'num_object_clones': 0, 'num_object_copies': 13968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3493, 'num_write_kb': 27941, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1242}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3522", 'reported_seq': 3562, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.477319+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:11.477319+0000', 'last_peered': '2024-09-11T04:57:11.477319+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:11.477319+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3522, 'log_dups_size': 0, 'ondisk_log_size': 3522, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28852224, 'num_objects': 3522, 'num_object_clones': 0, 'num_object_copies': 14088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3522, 'num_write_kb': 28176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3530", 'reported_seq': 3591, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.023897+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:11.023897+0000', 'last_peered': '2024-09-11T04:57:11.023897+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:11.023897+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3530, 'log_dups_size': 0, 'ondisk_log_size': 3530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28865696, 'num_objects': 3524, 'num_object_clones': 0, 'num_object_copies': 14096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3528, 'num_write_kb': 28204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1207}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3569", 'reported_seq': 3629, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.041728+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:11.041728+0000', 'last_peered': '2024-09-11T04:57:11.041728+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:11.041728+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3569, 'log_dups_size': 0, 'ondisk_log_size': 3569, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29193376, 'num_objects': 3564, 'num_object_clones': 0, 'num_object_copies': 14256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3567, 'num_write_kb': 28524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1203}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3607", 'reported_seq': 3659, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.507999+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:11.507999+0000', 'last_peered': '2024-09-11T04:57:11.507999+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:11.507999+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3607, 'log_dups_size': 0, 'ondisk_log_size': 3607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29477184, 'num_objects': 3599, 'num_object_clones': 0, 'num_object_copies': 14396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3604, 'num_write_kb': 28809, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1205}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3450", 'reported_seq': 3525, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:08.797309+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:08.797309+0000', 'last_peered': '2024-09-11T04:57:08.797309+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:08.797309+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3450, 'log_dups_size': 0, 'ondisk_log_size': 3450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28243104, 'num_objects': 3448, 'num_object_clones': 0, 'num_object_copies': 13792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3448, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3448, 'num_write_kb': 27582, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1130}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:15.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3504", 'reported_seq': 3558, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.610825+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:11.610825+0000', 'last_peered': '2024-09-11T04:57:11.610825+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:11.610825+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3504, 'log_dups_size': 0, 'ondisk_log_size': 3504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28659440, 'num_objects': 3499, 'num_object_clones': 0, 'num_object_copies': 13996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3501, 'num_write_kb': 27996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1176}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3459", 'reported_seq': 4532, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.555733+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:11.555733+0000', 'last_peered': '2024-09-11T04:57:11.555733+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:11.555733+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3459, 'log_dups_size': 0, 'ondisk_log_size': 3459, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28326480, 'num_objects': 3458, 'num_object_clones': 0, 'num_object_copies': 13832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1716, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3458, 'num_write_kb': 27663, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 520, 'num_bytes_recovered': 4251648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1742}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3514", 'reported_seq': 3587, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:08.785819+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:08.785819+0000', 'last_peered': '2024-09-11T04:57:08.785819+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:08.785819+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3514, 'log_dups_size': 0, 'ondisk_log_size': 3514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28777040, 'num_objects': 3513, 'num_object_clones': 0, 'num_object_copies': 14052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3513, 'num_write_kb': 28103, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1181}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3521", 'reported_seq': 3570, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.507232+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:11.507232+0000', 'last_peered': '2024-09-11T04:57:11.507232+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:11.507232+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3521, 'log_dups_size': 0, 'ondisk_log_size': 3521, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28808352, 'num_objects': 3517, 'num_object_clones': 0, 'num_object_copies': 14068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3519, 'num_write_kb': 28148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:15.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3477", 'reported_seq': 3554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:08.748273+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:08.748273+0000', 'last_peered': '2024-09-11T04:57:08.748273+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:08.748273+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3477, 'log_dups_size': 0, 'ondisk_log_size': 3477, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28456096, 'num_objects': 3474, 'num_object_clones': 0, 'num_object_copies': 13896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3475, 'num_write_kb': 27797, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1156}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:15.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:15.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3533", 'reported_seq': 3595, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:11.008583+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:11.008583+0000', 'last_peered': '2024-09-11T04:57:11.008583+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:11.008583+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3533, 'log_dups_size': 0, 'ondisk_log_size': 3533, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28898464, 'num_objects': 3528, 'num_object_clones': 0, 'num_object_copies': 14112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3531, 'num_write_kb': 28236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:15.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:15.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 92 20 57552 57532 4.88462 2.58594 0.0421736 0.0319741 2024-09-11T04:57:15.936 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:15.936 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36363/223372 objects degraded (16.279%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":162,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55843,"data_bytes":457425088,"bytes_used":1413992448,"bytes_avail":385133064192,"bytes_total":386547056640,"degraded_objects":36363,"degraded_total":223372,"degraded_ratio":0.16279121823684262,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":68929,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":827392,"num_keys_recovered":0,"write_bytes_sec":2427540,"read_op_per_sec":0,"write_op_per_sec":296},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-11T04:56:31.620416+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T04:57:15.954 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:15.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:16.635 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:16.636 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:16.663 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-11T04:57:15.634974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":460947648,"num_objects":56273,"num_object_clones":0,"num_object_copies":225092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56300,"num_write_kb":450282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56327,"ondisk_log_size":56327,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1391980,"kb_used_data":752172,"kb_used_omap":14,"kb_used_meta":639729,"kb_avail":376095380,"statfs":{"total":386547056640,"available":385121669120,"internally_reserved":0,"allocated":770224128,"data_stored":769540888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":655083062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,6,9,14,15,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32661504,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":827392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003823"},"pg_stats":[{"pgid":"1.f","version":"31'3455","reported_seq":3496,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.569276+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:11.569276+0000","last_peered":"2024-09-11T04:57:11.569276+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:11.569276+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3455,"log_dups_size":0,"ondisk_log_size":3455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28303360,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":27640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3501","reported_seq":3562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:10.932773+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:10.932773+0000","last_peered":"2024-09-11T04:57:10.932773+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:10.932773+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28644512,"num_objects":3497,"num_object_clones":0,"num_object_copies":13988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":27988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3404","reported_seq":3479,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:13.869452+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:13.869452+0000","last_peered":"2024-09-11T04:57:13.869452+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:13.869452+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3404,"log_dups_size":0,"ondisk_log_size":3404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27859536,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3403,"num_write_kb":27214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3450","reported_seq":3511,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.035183+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:11.035183+0000","last_peered":"2024-09-11T04:57:11.035183+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:11.035183+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28226720,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3495","reported_seq":3548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.595784+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:11.595784+0000","last_peered":"2024-09-11T04:57:11.595784+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:11.595784+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28603552,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":27941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3522","reported_seq":3562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.477319+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:11.477319+0000","last_peered":"2024-09-11T04:57:11.477319+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:11.477319+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3522,"log_dups_size":0,"ondisk_log_size":3522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28852224,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":28176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3530","reported_seq":3591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.023897+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:11.023897+0000","last_peered":"2024-09-11T04:57:11.023897+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:11.023897+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28865696,"num_objects":3524,"num_object_clones":0,"num_object_copies":14096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":28204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1207}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3569","reported_seq":3629,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.041728+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:11.041728+0000","last_peered":"2024-09-11T04:57:11.041728+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:11.041728+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29193376,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3567,"num_write_kb":28524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3607","reported_seq":3659,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.507999+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:11.507999+0000","last_peered":"2024-09-11T04:57:11.507999+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:11.507999+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3607,"log_dups_size":0,"ondisk_log_size":3607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29477184,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3562","reported_seq":3639,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:13.837202+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:13.837202+0000","last_peered":"2024-09-11T04:57:13.837202+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:13.837202+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3562,"log_dups_size":0,"ondisk_log_size":3562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29160608,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3560,"num_write_kb":28478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1242}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3504","reported_seq":3558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.610825+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:11.610825+0000","last_peered":"2024-09-11T04:57:11.610825+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:11.610825+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28659440,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":27996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3459","reported_seq":4532,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.555733+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:11.555733+0000","last_peered":"2024-09-11T04:57:11.555733+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:11.555733+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3459,"log_dups_size":0,"ondisk_log_size":3459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28326480,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3458,"num_write_kb":27663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1742}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3627","reported_seq":3700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:13.874992+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:13.874992+0000","last_peered":"2024-09-11T04:57:13.874992+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:13.874992+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29702736,"num_objects":3626,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":29007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1294}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3521","reported_seq":3570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.507232+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:11.507232+0000","last_peered":"2024-09-11T04:57:11.507232+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:11.507232+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28808352,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":28148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1230}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3588","reported_seq":3663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:13.850418+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:13.850418+0000","last_peered":"2024-09-11T04:57:13.850418+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:13.850418+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29365408,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3586,"num_write_kb":28685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1267}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3533","reported_seq":3595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:11.008583+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:11.008583+0000","last_peered":"2024-09-11T04:57:11.008583+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:11.008583+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28898464,"num_objects":3528,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":460947648,"num_objects":56273,"num_object_clones":0,"num_object_copies":225092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56300,"num_write_kb":450282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":4251648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":773623808,"data_stored":773623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56327,"ondisk_log_size":56327,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407996,"kb_used_data":223596,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93963844,"statfs":{"total":96636764160,"available":96218976256,"internally_reserved":0,"allocated":228962304,"data_stored":228791494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,7,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403028,"kb_used_data":227460,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93968812,"statfs":{"total":96636764160,"available":96224063488,"internally_reserved":0,"allocated":232919040,"data_stored":232748230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182236,"kb_used_data":78412,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94189604,"statfs":{"total":96636764160,"available":96450154496,"internally_reserved":0,"allocated":80293888,"data_stored":80123078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":106294994},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398720,"kb_used_data":222704,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93973120,"statfs":{"total":96636764160,"available":96228474880,"internally_reserved":0,"allocated":228048896,"data_stored":227878086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229019648,"data_stored":229019648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81457152,"data_stored":81457152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233279488,"data_stored":233279488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229867520,"data_stored":229867520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:16.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 93 20 57869 57849 4.85873 2.47656 0.0800749 0.0321426 2024-09-11T04:57:17.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 94 20 58235 58215 4.83746 2.85938 0.0706938 0.032285 2024-09-11T04:57:18.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 95 20 58594 58574 4.81606 2.80469 0.040243 0.0324287 2024-09-11T04:57:19.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:19.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 96 20 58956 58936 4.79536 2.82812 0.0437898 0.0325686 2024-09-11T04:57:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:20.296 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:20.317 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-11T04:57:19.636722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475185344,"num_objects":58011,"num_object_clones":0,"num_object_copies":232044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58038,"num_write_kb":464186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58065,"ondisk_log_size":58065,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1442496,"kb_used_data":780480,"kb_used_omap":14,"kb_used_meta":661937,"kb_avail":376044864,"statfs":{"total":386547056640,"available":385069940736,"internally_reserved":0,"allocated":799211520,"data_stored":798528280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":677824054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,8,4,10,20,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32145408,"num_objects":3924,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3924,"num_write_kb":31392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004660"},"pg_stats":[{"pgid":"1.f","version":"31'3537","reported_seq":3578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.524821+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:16.524821+0000","last_peered":"2024-09-11T04:57:16.524821+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:16.524821+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28975104,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":28296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3606","reported_seq":3667,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.023717+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:16.023717+0000","last_peered":"2024-09-11T04:57:16.023717+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:16.023717+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29504672,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3507","reported_seq":3582,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.800991+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:18.800991+0000","last_peered":"2024-09-11T04:57:18.800991+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:18.800991+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703312,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3506,"num_write_kb":28038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1284}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3548","reported_seq":3610,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:15.969793+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:15.969793+0000","last_peered":"2024-09-11T04:57:15.969793+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:15.969793+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29029536,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3546,"num_write_kb":28357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3612","reported_seq":3665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.547977+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:16.547977+0000","last_peered":"2024-09-11T04:57:16.547977+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:16.547977+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562016,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":28877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3630","reported_seq":3670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.572972+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:16.572972+0000","last_peered":"2024-09-11T04:57:16.572972+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:16.572972+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":29040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3630","reported_seq":3692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:15.911713+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:15.911713+0000","last_peered":"2024-09-11T04:57:15.911713+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:15.911713+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29684896,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":29004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1307}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3692","reported_seq":3752,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.092837+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:16.092837+0000","last_peered":"2024-09-11T04:57:16.092837+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:16.092837+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3692,"log_dups_size":0,"ondisk_log_size":3692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30200992,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":29508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3707","reported_seq":3762,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.622053+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:16.622053+0000","last_peered":"2024-09-11T04:57:16.622053+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:16.622053+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3707,"log_dups_size":0,"ondisk_log_size":3707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30296384,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3704,"num_write_kb":29609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3666","reported_seq":3743,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.711292+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:18.711292+0000","last_peered":"2024-09-11T04:57:18.711292+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:18.711292+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30012576,"num_objects":3664,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":29310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1346}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3629","reported_seq":3684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.630209+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:16.630209+0000","last_peered":"2024-09-11T04:57:16.630209+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:16.630209+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3629,"log_dups_size":0,"ondisk_log_size":3629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29683440,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":28996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3590","reported_seq":4768,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.692195+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:16.692195+0000","last_peered":"2024-09-11T04:57:16.692195+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:16.692195+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29399632,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1925}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3729","reported_seq":3802,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.833352+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:18.833352+0000","last_peered":"2024-09-11T04:57:18.833352+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:18.833352+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3729,"log_dups_size":0,"ondisk_log_size":3729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30538320,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":29823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3631","reported_seq":3680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.643850+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:16.643850+0000","last_peered":"2024-09-11T04:57:16.643850+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:16.643850+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3631,"log_dups_size":0,"ondisk_log_size":3631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29709472,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3629,"num_write_kb":29028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3700","reported_seq":3777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.849287+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:18.849287+0000","last_peered":"2024-09-11T04:57:18.849287+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:18.849287+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282912,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":29581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3651","reported_seq":3713,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:15.975737+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:15.975737+0000","last_peered":"2024-09-11T04:57:15.975737+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:15.975737+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29865120,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":475185344,"num_objects":58011,"num_object_clones":0,"num_object_copies":232044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58038,"num_write_kb":464186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802344960,"data_stored":802344960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58065,"ondisk_log_size":58065,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402624,"kb_used_data":230704,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93969216,"statfs":{"total":96636764160,"available":96224477184,"internally_reserved":0,"allocated":236240896,"data_stored":236070086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418216,"kb_used_data":234328,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93953624,"statfs":{"total":96636764160,"available":96208510976,"internally_reserved":0,"allocated":239951872,"data_stored":239781062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":211760,"kb_used_data":85728,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94160080,"statfs":{"total":96636764160,"available":96419921920,"internally_reserved":0,"allocated":87785472,"data_stored":87614662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129035986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409896,"kb_used_data":229720,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93961944,"statfs":{"total":96636764160,"available":96217030656,"internally_reserved":0,"allocated":235233280,"data_stored":235062470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236249088,"data_stored":236249088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88735744,"data_stored":88735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240324608,"data_stored":240324608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":237035520,"data_stored":237035520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:20.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:20.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 97 20 59269 59249 4.77113 2.44531 0.0425794 0.0327313 2024-09-11T04:57:20.997 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:20.997 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:21.019 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-11T04:57:19.636722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475185344,"num_objects":58011,"num_object_clones":0,"num_object_copies":232044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58038,"num_write_kb":464186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58065,"ondisk_log_size":58065,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1442496,"kb_used_data":780480,"kb_used_omap":14,"kb_used_meta":661937,"kb_avail":376044864,"statfs":{"total":386547056640,"available":385069940736,"internally_reserved":0,"allocated":799211520,"data_stored":798528280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":677824054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,8,4,10,20,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32145408,"num_objects":3924,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3924,"num_write_kb":31392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004660"},"pg_stats":[{"pgid":"1.f","version":"31'3537","reported_seq":3578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.524821+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:16.524821+0000","last_peered":"2024-09-11T04:57:16.524821+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:16.524821+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28975104,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":28296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3606","reported_seq":3667,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.023717+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:16.023717+0000","last_peered":"2024-09-11T04:57:16.023717+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:16.023717+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29504672,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3507","reported_seq":3582,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.800991+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:18.800991+0000","last_peered":"2024-09-11T04:57:18.800991+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:18.800991+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703312,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3506,"num_write_kb":28038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1284}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3548","reported_seq":3610,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:15.969793+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:15.969793+0000","last_peered":"2024-09-11T04:57:15.969793+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:15.969793+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29029536,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3546,"num_write_kb":28357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3612","reported_seq":3665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.547977+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:16.547977+0000","last_peered":"2024-09-11T04:57:16.547977+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:16.547977+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562016,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":28877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3630","reported_seq":3670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.572972+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:16.572972+0000","last_peered":"2024-09-11T04:57:16.572972+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:16.572972+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":29040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3630","reported_seq":3692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:15.911713+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:15.911713+0000","last_peered":"2024-09-11T04:57:15.911713+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:15.911713+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29684896,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":29004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1307}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3692","reported_seq":3752,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.092837+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:16.092837+0000","last_peered":"2024-09-11T04:57:16.092837+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:16.092837+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3692,"log_dups_size":0,"ondisk_log_size":3692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30200992,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":29508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3707","reported_seq":3762,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.622053+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:16.622053+0000","last_peered":"2024-09-11T04:57:16.622053+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:16.622053+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3707,"log_dups_size":0,"ondisk_log_size":3707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30296384,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3704,"num_write_kb":29609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3666","reported_seq":3743,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.711292+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:18.711292+0000","last_peered":"2024-09-11T04:57:18.711292+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:18.711292+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30012576,"num_objects":3664,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":29310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1346}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3629","reported_seq":3684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.630209+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:16.630209+0000","last_peered":"2024-09-11T04:57:16.630209+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:16.630209+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3629,"log_dups_size":0,"ondisk_log_size":3629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29683440,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":28996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3590","reported_seq":4768,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.692195+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:16.692195+0000","last_peered":"2024-09-11T04:57:16.692195+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:16.692195+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29399632,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1925}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3729","reported_seq":3802,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.833352+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:18.833352+0000","last_peered":"2024-09-11T04:57:18.833352+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:18.833352+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3729,"log_dups_size":0,"ondisk_log_size":3729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30538320,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":29823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3631","reported_seq":3680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.643850+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:16.643850+0000","last_peered":"2024-09-11T04:57:16.643850+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:16.643850+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3631,"log_dups_size":0,"ondisk_log_size":3631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29709472,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3629,"num_write_kb":29028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3700","reported_seq":3777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.849287+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:18.849287+0000","last_peered":"2024-09-11T04:57:18.849287+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:18.849287+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282912,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":29581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3651","reported_seq":3713,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:15.975737+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:15.975737+0000","last_peered":"2024-09-11T04:57:15.975737+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:15.975737+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29865120,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":475185344,"num_objects":58011,"num_object_clones":0,"num_object_copies":232044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58038,"num_write_kb":464186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802344960,"data_stored":802344960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58065,"ondisk_log_size":58065,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402624,"kb_used_data":230704,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93969216,"statfs":{"total":96636764160,"available":96224477184,"internally_reserved":0,"allocated":236240896,"data_stored":236070086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418216,"kb_used_data":234328,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93953624,"statfs":{"total":96636764160,"available":96208510976,"internally_reserved":0,"allocated":239951872,"data_stored":239781062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":211760,"kb_used_data":85728,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94160080,"statfs":{"total":96636764160,"available":96419921920,"internally_reserved":0,"allocated":87785472,"data_stored":87614662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129035986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409896,"kb_used_data":229720,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93961944,"statfs":{"total":96636764160,"available":96217030656,"internally_reserved":0,"allocated":235233280,"data_stored":235062470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236249088,"data_stored":236249088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88735744,"data_stored":88735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240324608,"data_stored":240324608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":237035520,"data_stored":237035520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:21.021 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:21.021 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3537", 'reported_seq': 3578, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.524821+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:16.524821+0000', 'last_peered': '2024-09-11T04:57:16.524821+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:16.524821+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3537, 'log_dups_size': 0, 'ondisk_log_size': 3537, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28975104, 'num_objects': 3537, 'num_object_clones': 0, 'num_object_copies': 14148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3537, 'num_write_kb': 28296, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1225}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:21.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:21.022 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3606", 'reported_seq': 3667, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.023717+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:16.023717+0000', 'last_peered': '2024-09-11T04:57:16.023717+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:16.023717+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3606, 'log_dups_size': 0, 'ondisk_log_size': 3606, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29504672, 'num_objects': 3602, 'num_object_clones': 0, 'num_object_copies': 14408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3604, 'num_write_kb': 28828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1323}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:21.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:21.022 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3507", 'reported_seq': 3582, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:18.800991+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:18.800991+0000', 'last_peered': '2024-09-11T04:57:18.800991+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:18.800991+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3507, 'log_dups_size': 0, 'ondisk_log_size': 3507, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28703312, 'num_objects': 3504, 'num_object_clones': 0, 'num_object_copies': 14016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3506, 'num_write_kb': 28038, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1284}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:21.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:21.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3548", 'reported_seq': 3610, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:15.969793+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:15.969793+0000', 'last_peered': '2024-09-11T04:57:15.969793+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:15.969793+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3548, 'log_dups_size': 0, 'ondisk_log_size': 3548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29029536, 'num_objects': 3544, 'num_object_clones': 0, 'num_object_copies': 14176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3546, 'num_write_kb': 28357, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:21.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:21.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3612", 'reported_seq': 3665, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.547977+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:16.547977+0000', 'last_peered': '2024-09-11T04:57:16.547977+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:16.547977+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3612, 'log_dups_size': 0, 'ondisk_log_size': 3612, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29562016, 'num_objects': 3609, 'num_object_clones': 0, 'num_object_copies': 14436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3610, 'num_write_kb': 28877, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1359}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:21.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:21.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3630", 'reported_seq': 3670, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.572972+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:16.572972+0000', 'last_peered': '2024-09-11T04:57:16.572972+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:16.572972+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3630, 'log_dups_size': 0, 'ondisk_log_size': 3630, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29736960, 'num_objects': 3630, 'num_object_clones': 0, 'num_object_copies': 14520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3630, 'num_write_kb': 29040, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:21.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:21.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3630", 'reported_seq': 3692, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:15.911713+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:15.911713+0000', 'last_peered': '2024-09-11T04:57:15.911713+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:15.911713+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3630, 'log_dups_size': 0, 'ondisk_log_size': 3630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29684896, 'num_objects': 3624, 'num_object_clones': 0, 'num_object_copies': 14496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3628, 'num_write_kb': 29004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1307}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:21.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:21.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3692", 'reported_seq': 3752, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.092837+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:16.092837+0000', 'last_peered': '2024-09-11T04:57:16.092837+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:16.092837+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3692, 'log_dups_size': 0, 'ondisk_log_size': 3692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30200992, 'num_objects': 3687, 'num_object_clones': 0, 'num_object_copies': 14748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3690, 'num_write_kb': 29508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1326}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:21.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:21.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3707", 'reported_seq': 3762, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.622053+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:16.622053+0000', 'last_peered': '2024-09-11T04:57:16.622053+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:16.622053+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3707, 'log_dups_size': 0, 'ondisk_log_size': 3707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30296384, 'num_objects': 3699, 'num_object_clones': 0, 'num_object_copies': 14796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3704, 'num_write_kb': 29609, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1305}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:21.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:21.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3666", 'reported_seq': 3743, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:18.711292+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:18.711292+0000', 'last_peered': '2024-09-11T04:57:18.711292+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:18.711292+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3666, 'log_dups_size': 0, 'ondisk_log_size': 3666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30012576, 'num_objects': 3664, 'num_object_clones': 0, 'num_object_copies': 14656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3664, 'num_write_kb': 29310, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1346}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:21.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:21.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3629", 'reported_seq': 3684, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.630209+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:16.630209+0000', 'last_peered': '2024-09-11T04:57:16.630209+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:16.630209+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3629, 'log_dups_size': 0, 'ondisk_log_size': 3629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29683440, 'num_objects': 3624, 'num_object_clones': 0, 'num_object_copies': 14496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3626, 'num_write_kb': 28996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1301}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:21.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:21.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3590", 'reported_seq': 4768, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.692195+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:16.692195+0000', 'last_peered': '2024-09-11T04:57:16.692195+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:16.692195+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3590, 'log_dups_size': 0, 'ondisk_log_size': 3590, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29399632, 'num_objects': 3589, 'num_object_clones': 0, 'num_object_copies': 14356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1664, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3589, 'num_write_kb': 28711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 572, 'num_bytes_recovered': 4677632, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1925}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:21.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:21.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3729", 'reported_seq': 3802, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:18.833352+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:18.833352+0000', 'last_peered': '2024-09-11T04:57:18.833352+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:18.833352+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3729, 'log_dups_size': 0, 'ondisk_log_size': 3729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30538320, 'num_objects': 3728, 'num_object_clones': 0, 'num_object_copies': 14912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3728, 'num_write_kb': 29823, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:21.027 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:21.027 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3631", 'reported_seq': 3680, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:16.643850+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:16.643850+0000', 'last_peered': '2024-09-11T04:57:16.643850+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:16.643850+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3631, 'log_dups_size': 0, 'ondisk_log_size': 3631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29709472, 'num_objects': 3627, 'num_object_clones': 0, 'num_object_copies': 14508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3629, 'num_write_kb': 29028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:21.027 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:21.027 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3700", 'reported_seq': 3777, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:18.849287+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:18.849287+0000', 'last_peered': '2024-09-11T04:57:18.849287+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:18.849287+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3700, 'log_dups_size': 0, 'ondisk_log_size': 3700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30282912, 'num_objects': 3697, 'num_object_clones': 0, 'num_object_copies': 14788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3698, 'num_write_kb': 29581, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:21.028 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:21.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3651", 'reported_seq': 3713, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:15.975737+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:15.975737+0000', 'last_peered': '2024-09-11T04:57:15.975737+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:15.975737+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3651, 'log_dups_size': 0, 'ondisk_log_size': 3651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29865120, 'num_objects': 3646, 'num_object_clones': 0, 'num_object_copies': 14584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3649, 'num_write_kb': 29180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:21.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:21.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 98 20 59584 59564 4.74756 2.46094 0.0796311 0.0328945 2024-09-11T04:57:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:21.783 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36311/232044 objects degraded (15.648%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":168,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58011,"data_bytes":475185344,"bytes_used":1477115904,"bytes_avail":385069940736,"bytes_total":386547056640,"degraded_objects":36311,"degraded_total":232044,"degraded_ratio":0.15648325317612177,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":67557,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"write_bytes_sec":2677743,"read_op_per_sec":0,"write_op_per_sec":326},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:57:21.807 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:21.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:22.465 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:22.465 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:22.492 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-11T04:57:21.637527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":479879360,"num_objects":58584,"num_object_clones":0,"num_object_copies":234336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58611,"num_write_kb":468770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58638,"ondisk_log_size":58638,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1472796,"kb_used_data":788572,"kb_used_omap":14,"kb_used_meta":684145,"kb_avail":376014564,"statfs":{"total":386547056640,"available":385038913536,"internally_reserved":0,"allocated":807497728,"data_stored":806814488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":700565046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,5,15,23,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33005568,"num_objects":4029,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":32232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004966"},"pg_stats":[{"pgid":"1.f","version":"31'3537","reported_seq":3578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.524821+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:16.524821+0000","last_peered":"2024-09-11T04:57:16.524821+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:16.524821+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28975104,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":28296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3732","reported_seq":3793,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.133484+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:21.133484+0000","last_peered":"2024-09-11T04:57:21.133484+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:21.133484+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30536864,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3507","reported_seq":3582,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.800991+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:18.800991+0000","last_peered":"2024-09-11T04:57:18.800991+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:18.800991+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703312,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3506,"num_write_kb":28038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1284}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3651","reported_seq":3712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.113236+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:21.113236+0000","last_peered":"2024-09-11T04:57:21.113236+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:21.113236+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873312,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3612","reported_seq":3665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.547977+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:16.547977+0000","last_peered":"2024-09-11T04:57:16.547977+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:16.547977+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562016,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":28877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3630","reported_seq":3670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.572972+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:16.572972+0000","last_peered":"2024-09-11T04:57:16.572972+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:16.572972+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":29040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3748","reported_seq":3810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.122021+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:21.122021+0000","last_peered":"2024-09-11T04:57:21.122021+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:21.122021+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3748,"log_dups_size":0,"ondisk_log_size":3748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30651552,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3809","reported_seq":3870,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.129588+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:21.129588+0000","last_peered":"2024-09-11T04:57:21.129588+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:21.129588+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31159456,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":30444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3707","reported_seq":3762,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.622053+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:16.622053+0000","last_peered":"2024-09-11T04:57:16.622053+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:16.622053+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3707,"log_dups_size":0,"ondisk_log_size":3707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30296384,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3704,"num_write_kb":29609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3666","reported_seq":3743,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.711292+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:18.711292+0000","last_peered":"2024-09-11T04:57:18.711292+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:18.711292+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30012576,"num_objects":3664,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":29310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1346}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3629","reported_seq":3684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.630209+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:16.630209+0000","last_peered":"2024-09-11T04:57:16.630209+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:16.630209+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3629,"log_dups_size":0,"ondisk_log_size":3629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29683440,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":28996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3590","reported_seq":4768,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.692195+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:16.692195+0000","last_peered":"2024-09-11T04:57:16.692195+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:16.692195+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29399632,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1925}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3729","reported_seq":3802,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.833352+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:18.833352+0000","last_peered":"2024-09-11T04:57:18.833352+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:18.833352+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3729,"log_dups_size":0,"ondisk_log_size":3729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30538320,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":29823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3631","reported_seq":3680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:16.643850+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:16.643850+0000","last_peered":"2024-09-11T04:57:16.643850+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:16.643850+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3631,"log_dups_size":0,"ondisk_log_size":3631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29709472,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3629,"num_write_kb":29028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3700","reported_seq":3777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:18.849287+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:18.849287+0000","last_peered":"2024-09-11T04:57:18.849287+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:18.849287+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282912,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":29581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3760","reported_seq":3822,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.064182+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:21.064182+0000","last_peered":"2024-09-11T04:57:21.064182+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:21.064182+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30758048,"num_objects":3755,"num_object_clones":0,"num_object_copies":15020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3758,"num_write_kb":30052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":479879360,"num_objects":58584,"num_object_clones":0,"num_object_copies":234336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58611,"num_write_kb":468770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":809447424,"data_stored":809447424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58638,"ondisk_log_size":58638,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402624,"kb_used_data":230704,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93969216,"statfs":{"total":96636764160,"available":96224477184,"internally_reserved":0,"allocated":236240896,"data_stored":236070086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418216,"kb_used_data":234328,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93953624,"statfs":{"total":96636764160,"available":96208510976,"internally_reserved":0,"allocated":239951872,"data_stored":239781062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":211760,"kb_used_data":85728,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94160080,"statfs":{"total":96636764160,"available":96419921920,"internally_reserved":0,"allocated":87785472,"data_stored":87614662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129035986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440196,"kb_used_data":237812,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93931644,"statfs":{"total":96636764160,"available":96186003456,"internally_reserved":0,"allocated":243519488,"data_stored":243348678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,6,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243351552,"data_stored":243351552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88735744,"data_stored":88735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240324608,"data_stored":240324608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":237035520,"data_stored":237035520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:22.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 99 20 59909 59889 4.72525 2.53906 0.0322603 0.0330534 2024-09-11T04:57:23.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:57:23.700918+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0331847 2024-09-11T04:57:23.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:57:23.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 100 20 60271 60251 4.70628 2.82812 0.0332527 0.0331847 2024-09-11T04:57:24.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 101 20 60626 60606 4.68714 2.77344 0.0469692 0.0333194 2024-09-11T04:57:25.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:25.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 102 20 60982 60962 4.66845 2.78125 0.0583331 0.0334528 2024-09-11T04:57:26.134 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:26.135 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:26.152 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-11T04:57:25.639517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489562304,"num_objects":59766,"num_object_clones":0,"num_object_copies":239064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59793,"num_write_kb":478226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59820,"ondisk_log_size":59820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1529776,"kb_used_data":809456,"kb_used_omap":14,"kb_used_meta":720241,"kb_avail":375957584,"statfs":{"total":386547056640,"available":384980566016,"internally_reserved":0,"allocated":828882944,"data_stored":828199704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":737527350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,8,3,16,19,16,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32137216,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":31384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005369"},"pg_stats":[{"pgid":"1.f","version":"31'3643","reported_seq":3683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.679821+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:21.679821+0000","last_peered":"2024-09-11T04:57:21.679821+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:21.679821+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29843456,"num_objects":3643,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":29144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3732","reported_seq":3793,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.133484+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:21.133484+0000","last_peered":"2024-09-11T04:57:21.133484+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:21.133484+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30536864,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3608","reported_seq":3683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.844844+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:23.844844+0000","last_peered":"2024-09-11T04:57:23.844844+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:23.844844+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3608,"log_dups_size":0,"ondisk_log_size":3608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29530704,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":28846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3651","reported_seq":3712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.113236+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:21.113236+0000","last_peered":"2024-09-11T04:57:21.113236+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:21.113236+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873312,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3717","reported_seq":3769,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.702781+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:21.702781+0000","last_peered":"2024-09-11T04:57:21.702781+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:21.702781+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30422176,"num_objects":3714,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3715,"num_write_kb":29717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3757","reported_seq":3797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.769029+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:21.769029+0000","last_peered":"2024-09-11T04:57:21.769029+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:21.769029+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30777344,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3757,"num_write_kb":30056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3748","reported_seq":3810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.122021+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:21.122021+0000","last_peered":"2024-09-11T04:57:21.122021+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:21.122021+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3748,"log_dups_size":0,"ondisk_log_size":3748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30651552,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3809","reported_seq":3870,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.129588+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:21.129588+0000","last_peered":"2024-09-11T04:57:21.129588+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:21.129588+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31159456,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":30444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3823","reported_seq":3878,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.645378+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:21.645378+0000","last_peered":"2024-09-11T04:57:21.645378+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:21.645378+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3823,"log_dups_size":0,"ondisk_log_size":3823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31246656,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":30537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3781","reported_seq":3858,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.834056+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:23.834056+0000","last_peered":"2024-09-11T04:57:23.834056+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:23.834056+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30954656,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":30230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3736","reported_seq":3791,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.644677+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:21.644677+0000","last_peered":"2024-09-11T04:57:21.644677+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:21.644677+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30559984,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3733,"num_write_kb":29852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1408}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3678","reported_seq":4973,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.811436+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:21.811436+0000","last_peered":"2024-09-11T04:57:21.811436+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:21.811436+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30120528,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3677,"num_write_kb":29415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3821","reported_seq":3893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.930616+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:23.930616+0000","last_peered":"2024-09-11T04:57:23.930616+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:23.930616+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31291984,"num_objects":3820,"num_object_clones":0,"num_object_copies":15280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":30559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1488}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3751","reported_seq":3799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.752209+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:21.752209+0000","last_peered":"2024-09-11T04:57:21.752209+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:21.752209+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692512,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3749,"num_write_kb":29988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1460}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3805","reported_seq":3882,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.886358+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:23.886358+0000","last_peered":"2024-09-11T04:57:23.886358+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:23.886358+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3760","reported_seq":3822,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.064182+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:21.064182+0000","last_peered":"2024-09-11T04:57:21.064182+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:21.064182+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30758048,"num_objects":3755,"num_object_clones":0,"num_object_copies":15020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3758,"num_write_kb":30052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":489562304,"num_objects":59766,"num_object_clones":0,"num_object_copies":239064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59793,"num_write_kb":478226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":830947328,"data_stored":830947328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59820,"ondisk_log_size":59820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431844,"kb_used_data":237716,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93939996,"statfs":{"total":96636764160,"available":96194555904,"internally_reserved":0,"allocated":243421184,"data_stored":243250374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":198766288},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,0,2,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447032,"kb_used_data":240936,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":93924808,"statfs":{"total":96636764160,"available":96179003392,"internally_reserved":0,"allocated":246718464,"data_stored":246547654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":210704,"kb_used_data":92992,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94161136,"statfs":{"total":96636764160,"available":96421003264,"internally_reserved":0,"allocated":95223808,"data_stored":95052998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120516306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,6,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440196,"kb_used_data":237812,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93931644,"statfs":{"total":96636764160,"available":96186003456,"internally_reserved":0,"allocated":243519488,"data_stored":243348678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,6,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243351552,"data_stored":243351552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96178176,"data_stored":96178176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247201792,"data_stored":247201792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":244215808,"data_stored":244215808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:26.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:26.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 103 20 61297 61277 4.64702 2.46094 0.0476764 0.0336059 2024-09-11T04:57:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:26.773 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:26.796 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-11T04:57:25.639517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489562304,"num_objects":59766,"num_object_clones":0,"num_object_copies":239064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59793,"num_write_kb":478226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59820,"ondisk_log_size":59820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1529776,"kb_used_data":809456,"kb_used_omap":14,"kb_used_meta":720241,"kb_avail":375957584,"statfs":{"total":386547056640,"available":384980566016,"internally_reserved":0,"allocated":828882944,"data_stored":828199704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":737527350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,8,3,16,19,16,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32137216,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":31384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005369"},"pg_stats":[{"pgid":"1.f","version":"31'3643","reported_seq":3683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.679821+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:21.679821+0000","last_peered":"2024-09-11T04:57:21.679821+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:21.679821+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29843456,"num_objects":3643,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":29144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3732","reported_seq":3793,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.133484+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:21.133484+0000","last_peered":"2024-09-11T04:57:21.133484+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:21.133484+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30536864,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3608","reported_seq":3683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.844844+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:23.844844+0000","last_peered":"2024-09-11T04:57:23.844844+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:23.844844+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3608,"log_dups_size":0,"ondisk_log_size":3608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29530704,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":28846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3651","reported_seq":3712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.113236+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:21.113236+0000","last_peered":"2024-09-11T04:57:21.113236+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:21.113236+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873312,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3717","reported_seq":3769,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.702781+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:21.702781+0000","last_peered":"2024-09-11T04:57:21.702781+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:21.702781+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30422176,"num_objects":3714,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3715,"num_write_kb":29717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3757","reported_seq":3797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.769029+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:21.769029+0000","last_peered":"2024-09-11T04:57:21.769029+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:21.769029+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30777344,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3757,"num_write_kb":30056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3748","reported_seq":3810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.122021+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:21.122021+0000","last_peered":"2024-09-11T04:57:21.122021+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:21.122021+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3748,"log_dups_size":0,"ondisk_log_size":3748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30651552,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3809","reported_seq":3870,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.129588+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:21.129588+0000","last_peered":"2024-09-11T04:57:21.129588+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:21.129588+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31159456,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":30444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3823","reported_seq":3878,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.645378+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:21.645378+0000","last_peered":"2024-09-11T04:57:21.645378+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:21.645378+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3823,"log_dups_size":0,"ondisk_log_size":3823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31246656,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":30537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3781","reported_seq":3858,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.834056+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:23.834056+0000","last_peered":"2024-09-11T04:57:23.834056+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:23.834056+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30954656,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":30230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3736","reported_seq":3791,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.644677+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:21.644677+0000","last_peered":"2024-09-11T04:57:21.644677+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:21.644677+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30559984,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3733,"num_write_kb":29852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1408}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3678","reported_seq":4973,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.811436+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:21.811436+0000","last_peered":"2024-09-11T04:57:21.811436+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:21.811436+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30120528,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3677,"num_write_kb":29415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3821","reported_seq":3893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.930616+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:23.930616+0000","last_peered":"2024-09-11T04:57:23.930616+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:23.930616+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31291984,"num_objects":3820,"num_object_clones":0,"num_object_copies":15280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":30559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1488}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3751","reported_seq":3799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.752209+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:21.752209+0000","last_peered":"2024-09-11T04:57:21.752209+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:21.752209+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692512,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3749,"num_write_kb":29988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1460}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3805","reported_seq":3882,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.886358+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:23.886358+0000","last_peered":"2024-09-11T04:57:23.886358+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:23.886358+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3760","reported_seq":3822,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:21.064182+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:21.064182+0000","last_peered":"2024-09-11T04:57:21.064182+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:21.064182+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30758048,"num_objects":3755,"num_object_clones":0,"num_object_copies":15020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3758,"num_write_kb":30052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":489562304,"num_objects":59766,"num_object_clones":0,"num_object_copies":239064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59793,"num_write_kb":478226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":830947328,"data_stored":830947328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59820,"ondisk_log_size":59820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431844,"kb_used_data":237716,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93939996,"statfs":{"total":96636764160,"available":96194555904,"internally_reserved":0,"allocated":243421184,"data_stored":243250374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":198766288},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,0,2,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447032,"kb_used_data":240936,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":93924808,"statfs":{"total":96636764160,"available":96179003392,"internally_reserved":0,"allocated":246718464,"data_stored":246547654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":210704,"kb_used_data":92992,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94161136,"statfs":{"total":96636764160,"available":96421003264,"internally_reserved":0,"allocated":95223808,"data_stored":95052998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120516306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,6,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440196,"kb_used_data":237812,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93931644,"statfs":{"total":96636764160,"available":96186003456,"internally_reserved":0,"allocated":243519488,"data_stored":243348678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,6,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243351552,"data_stored":243351552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96178176,"data_stored":96178176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247201792,"data_stored":247201792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":244215808,"data_stored":244215808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:26.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:26.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3643", 'reported_seq': 3683, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.679821+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:21.679821+0000', 'last_peered': '2024-09-11T04:57:21.679821+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:21.679821+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3643, 'log_dups_size': 0, 'ondisk_log_size': 3643, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29843456, 'num_objects': 3643, 'num_object_clones': 0, 'num_object_copies': 14572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3643, 'num_write_kb': 29144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1331}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:26.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:26.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3732", 'reported_seq': 3793, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.133484+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:21.133484+0000', 'last_peered': '2024-09-11T04:57:21.133484+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:21.133484+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3732, 'log_dups_size': 0, 'ondisk_log_size': 3732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30536864, 'num_objects': 3728, 'num_object_clones': 0, 'num_object_copies': 14912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3730, 'num_write_kb': 29836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3608", 'reported_seq': 3683, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:23.844844+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:23.844844+0000', 'last_peered': '2024-09-11T04:57:23.844844+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:23.844844+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3608, 'log_dups_size': 0, 'ondisk_log_size': 3608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29530704, 'num_objects': 3605, 'num_object_clones': 0, 'num_object_copies': 14420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3607, 'num_write_kb': 28846, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1385}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3651", 'reported_seq': 3712, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.113236+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:21.113236+0000', 'last_peered': '2024-09-11T04:57:21.113236+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:21.113236+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3651, 'log_dups_size': 0, 'ondisk_log_size': 3651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29873312, 'num_objects': 3647, 'num_object_clones': 0, 'num_object_copies': 14588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3649, 'num_write_kb': 29181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1369}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3717", 'reported_seq': 3769, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.702781+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:21.702781+0000', 'last_peered': '2024-09-11T04:57:21.702781+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:21.702781+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3717, 'log_dups_size': 0, 'ondisk_log_size': 3717, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30422176, 'num_objects': 3714, 'num_object_clones': 0, 'num_object_copies': 14856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3715, 'num_write_kb': 29717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1464}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3757", 'reported_seq': 3797, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.769029+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:21.769029+0000', 'last_peered': '2024-09-11T04:57:21.769029+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:21.769029+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3757, 'log_dups_size': 0, 'ondisk_log_size': 3757, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30777344, 'num_objects': 3757, 'num_object_clones': 0, 'num_object_copies': 15028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3757, 'num_write_kb': 30056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1436}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:26.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3748", 'reported_seq': 3810, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.122021+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:21.122021+0000', 'last_peered': '2024-09-11T04:57:21.122021+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:21.122021+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3748, 'log_dups_size': 0, 'ondisk_log_size': 3748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30651552, 'num_objects': 3742, 'num_object_clones': 0, 'num_object_copies': 14968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3746, 'num_write_kb': 29948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1425}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3809", 'reported_seq': 3870, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.129588+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:21.129588+0000', 'last_peered': '2024-09-11T04:57:21.129588+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:21.129588+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3809, 'log_dups_size': 0, 'ondisk_log_size': 3809, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31159456, 'num_objects': 3804, 'num_object_clones': 0, 'num_object_copies': 15216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3807, 'num_write_kb': 30444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1443}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3823", 'reported_seq': 3878, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.645378+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:21.645378+0000', 'last_peered': '2024-09-11T04:57:21.645378+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:21.645378+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3823, 'log_dups_size': 0, 'ondisk_log_size': 3823, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31246656, 'num_objects': 3815, 'num_object_clones': 0, 'num_object_copies': 15260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3820, 'num_write_kb': 30537, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1421}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3781", 'reported_seq': 3858, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:23.834056+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:23.834056+0000', 'last_peered': '2024-09-11T04:57:23.834056+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:23.834056+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3781, 'log_dups_size': 0, 'ondisk_log_size': 3781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30954656, 'num_objects': 3779, 'num_object_clones': 0, 'num_object_copies': 15116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3779, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3779, 'num_write_kb': 30230, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1461}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3736", 'reported_seq': 3791, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.644677+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:21.644677+0000', 'last_peered': '2024-09-11T04:57:21.644677+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:21.644677+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3736, 'log_dups_size': 0, 'ondisk_log_size': 3736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30559984, 'num_objects': 3731, 'num_object_clones': 0, 'num_object_copies': 14924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3733, 'num_write_kb': 29852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1408}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3678", 'reported_seq': 4973, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.811436+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:21.811436+0000', 'last_peered': '2024-09-11T04:57:21.811436+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:21.811436+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3678, 'log_dups_size': 0, 'ondisk_log_size': 3678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30120528, 'num_objects': 3677, 'num_object_clones': 0, 'num_object_copies': 14708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1605, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3677, 'num_write_kb': 29415, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 631, 'num_bytes_recovered': 5160960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2072}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:26.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:26.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3821", 'reported_seq': 3893, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:23.930616+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:23.930616+0000', 'last_peered': '2024-09-11T04:57:23.930616+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:23.930616+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3821, 'log_dups_size': 0, 'ondisk_log_size': 3821, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31291984, 'num_objects': 3820, 'num_object_clones': 0, 'num_object_copies': 15280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3820, 'num_write_kb': 30559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:26.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:26.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3751", 'reported_seq': 3799, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.752209+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:21.752209+0000', 'last_peered': '2024-09-11T04:57:21.752209+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:21.752209+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3751, 'log_dups_size': 0, 'ondisk_log_size': 3751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30692512, 'num_objects': 3747, 'num_object_clones': 0, 'num_object_copies': 14988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3749, 'num_write_kb': 29988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1460}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:26.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:26.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3805", 'reported_seq': 3882, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:23.886358+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:23.886358+0000', 'last_peered': '2024-09-11T04:57:23.886358+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:23.886358+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3805, 'log_dups_size': 0, 'ondisk_log_size': 3805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31143072, 'num_objects': 3802, 'num_object_clones': 0, 'num_object_copies': 15208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3803, 'num_write_kb': 30421, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1484}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:26.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:26.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3760", 'reported_seq': 3822, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:21.064182+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:21.064182+0000', 'last_peered': '2024-09-11T04:57:21.064182+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:21.064182+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3760, 'log_dups_size': 0, 'ondisk_log_size': 3760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30758048, 'num_objects': 3755, 'num_object_clones': 0, 'num_object_copies': 15020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3758, 'num_write_kb': 30052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:26.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:27.598 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36252/239064 objects degraded (15.164%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":174,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59766,"data_bytes":489562304,"bytes_used":1566490624,"bytes_avail":384980566016,"bytes_total":386547056640,"degraded_objects":36252,"degraded_total":239064,"degraded_ratio":0.15164140146571628,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":75742,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"write_bytes_sec":2676903,"read_op_per_sec":0,"write_op_per_sec":326},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:57:27.622 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:27.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:27.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 104 20 61615 61595 4.62623 2.48438 0.0385885 0.0337583 2024-09-11T04:57:28.296 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:28.318 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-11T04:57:27.640381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":500113600,"num_objects":61054,"num_object_clones":0,"num_object_copies":244216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61081,"num_write_kb":488530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61108,"ondisk_log_size":61108,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1582296,"kb_used_data":830040,"kb_used_omap":14,"kb_used_meta":752177,"kb_avail":375905064,"statfs":{"total":386547056640,"available":384926785536,"internally_reserved":0,"allocated":849960960,"data_stored":849277720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":770229814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,9,15,13,17,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39165952,"num_objects":4781,"num_object_clones":0,"num_object_copies":19124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":38248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005407"},"pg_stats":[{"pgid":"1.f","version":"31'3744","reported_seq":3785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.852705+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:26.852705+0000","last_peered":"2024-09-11T04:57:26.852705+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:26.852705+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3744,"log_dups_size":0,"ondisk_log_size":3744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3831","reported_seq":3892,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.153625+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:26.153625+0000","last_peered":"2024-09-11T04:57:26.153625+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:26.153625+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3831,"log_dups_size":0,"ondisk_log_size":3831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31347872,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3829,"num_write_kb":30628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3608","reported_seq":3683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.844844+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:23.844844+0000","last_peered":"2024-09-11T04:57:23.844844+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:23.844844+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3608,"log_dups_size":0,"ondisk_log_size":3608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29530704,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":28846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3759","reported_seq":3820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.128536+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:26.128536+0000","last_peered":"2024-09-11T04:57:26.128536+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:26.128536+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30758048,"num_objects":3755,"num_object_clones":0,"num_object_copies":15020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3757,"num_write_kb":30045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3855","reported_seq":3908,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.742150+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:26.742150+0000","last_peered":"2024-09-11T04:57:26.742150+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:26.742150+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31552672,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":30821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3855","reported_seq":3895,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.837984+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:26.837984+0000","last_peered":"2024-09-11T04:57:26.837984+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:26.837984+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31580160,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":30840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1534}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3843","reported_seq":3904,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.046879+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:26.046879+0000","last_peered":"2024-09-11T04:57:26.046879+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:26.046879+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3843,"log_dups_size":0,"ondisk_log_size":3843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31429792,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3841,"num_write_kb":30708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1520}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3943","reported_seq":4004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.189422+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:26.189422+0000","last_peered":"2024-09-11T04:57:26.189422+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:26.189422+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32257184,"num_objects":3938,"num_object_clones":0,"num_object_copies":15752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":31516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3924","reported_seq":3979,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.777291+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:26.777291+0000","last_peered":"2024-09-11T04:57:26.777291+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:26.777291+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32074048,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3921,"num_write_kb":31345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3781","reported_seq":3858,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.834056+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:23.834056+0000","last_peered":"2024-09-11T04:57:23.834056+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:23.834056+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30954656,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":30230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3845","reported_seq":3900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.805160+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:26.805160+0000","last_peered":"2024-09-11T04:57:26.805160+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:26.805160+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31452912,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":30724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3763","reported_seq":5171,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.855733+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:26.855733+0000","last_peered":"2024-09-11T04:57:26.855733+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:26.855733+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3763,"log_dups_size":0,"ondisk_log_size":3763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30816848,"num_objects":3762,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3762,"num_write_kb":30095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2213}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3821","reported_seq":3893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.930616+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:23.930616+0000","last_peered":"2024-09-11T04:57:23.930616+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:23.930616+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31291984,"num_objects":3820,"num_object_clones":0,"num_object_copies":15280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3820,"num_write_kb":30559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1488}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3867","reported_seq":3916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.843528+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:26.843528+0000","last_peered":"2024-09-11T04:57:26.843528+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:26.843528+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642784,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1576}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3805","reported_seq":3882,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:23.886358+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:23.886358+0000","last_peered":"2024-09-11T04:57:23.886358+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:23.886358+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3864","reported_seq":3926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.078730+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:26.078730+0000","last_peered":"2024-09-11T04:57:26.078730+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:26.078730+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31610016,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3862,"num_write_kb":30884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":500113600,"num_objects":61054,"num_object_clones":0,"num_object_copies":244216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61081,"num_write_kb":488530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":852422656,"data_stored":852422656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61108,"ondisk_log_size":61108,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460776,"kb_used_data":244440,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93911064,"statfs":{"total":96636764160,"available":96164929536,"internally_reserved":0,"allocated":250306560,"data_stored":250135750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,5,1,4,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447032,"kb_used_data":240936,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":93924808,"statfs":{"total":96636764160,"available":96179003392,"internally_reserved":0,"allocated":246718464,"data_stored":246547654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221896,"kb_used_data":100024,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94149944,"statfs":{"total":96636764160,"available":96409542656,"internally_reserved":0,"allocated":102424576,"data_stored":102253766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124776146},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":452592,"kb_used_data":244640,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93919248,"statfs":{"total":96636764160,"available":96173309952,"internally_reserved":0,"allocated":250511360,"data_stored":250340550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212922063},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":250458112,"data_stored":250458112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103415808,"data_stored":103415808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247201792,"data_stored":247201792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251346944,"data_stored":251346944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:28.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 105 20 61952 61932 4.60724 2.63281 0.0595544 0.0338983 2024-09-11T04:57:29.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 106 20 62308 62288 4.59001 2.78125 0.0380181 0.034028 2024-09-11T04:57:30.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 107 20 62659 62639 4.57274 2.74219 0.0306295 0.0341526 2024-09-11T04:57:31.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:31.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 108 20 62996 62976 4.55478 2.63281 0.0462695 0.0342889 2024-09-11T04:57:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:31.974 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:32.005 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-11T04:57:31.641969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508027072,"num_objects":62020,"num_object_clones":0,"num_object_copies":248080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62047,"num_write_kb":496258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62074,"ondisk_log_size":62074,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1609636,"kb_used_data":843488,"kb_used_omap":14,"kb_used_meta":766065,"kb_avail":375877724,"statfs":{"total":386547056640,"available":384898789376,"internally_reserved":0,"allocated":863731712,"data_stored":863048472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":784451126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,14,12,10,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32841728,"num_objects":4009,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":32072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005246"},"pg_stats":[{"pgid":"1.f","version":"31'3744","reported_seq":3785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.852705+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:26.852705+0000","last_peered":"2024-09-11T04:57:26.852705+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:26.852705+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3744,"log_dups_size":0,"ondisk_log_size":3744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3938","reported_seq":3999,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.120410+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:31.120410+0000","last_peered":"2024-09-11T04:57:31.120410+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:31.120410+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32224416,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3719","reported_seq":3792,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.939080+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:28.939080+0000","last_peered":"2024-09-11T04:57:28.939080+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:28.939080+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30440016,"num_objects":3716,"num_object_clones":0,"num_object_copies":14864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":29734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3867","reported_seq":3930,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.088228+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:31.088228+0000","last_peered":"2024-09-11T04:57:31.088228+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:31.088228+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642784,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3855","reported_seq":3908,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.742150+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:26.742150+0000","last_peered":"2024-09-11T04:57:26.742150+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:26.742150+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31552672,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":30821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3855","reported_seq":3895,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.837984+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:26.837984+0000","last_peered":"2024-09-11T04:57:26.837984+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:26.837984+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31580160,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":30840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1534}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3957","reported_seq":4019,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.181862+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:31.181862+0000","last_peered":"2024-09-11T04:57:31.181862+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:31.181862+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363680,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3955,"num_write_kb":31620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1634}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4046","reported_seq":4108,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.271765+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:31.271765+0000","last_peered":"2024-09-11T04:57:31.271765+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:31.271765+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":32340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3924","reported_seq":3979,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.777291+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:26.777291+0000","last_peered":"2024-09-11T04:57:26.777291+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:26.777291+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32074048,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3921,"num_write_kb":31345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3884","reported_seq":3960,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.899015+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:28.899015+0000","last_peered":"2024-09-11T04:57:28.899015+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:28.899015+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31798432,"num_objects":3882,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1564}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3845","reported_seq":3900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.805160+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:26.805160+0000","last_peered":"2024-09-11T04:57:26.805160+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:26.805160+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31452912,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":30724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3763","reported_seq":5171,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.855733+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:26.855733+0000","last_peered":"2024-09-11T04:57:26.855733+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:26.855733+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3763,"log_dups_size":0,"ondisk_log_size":3763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30816848,"num_objects":3762,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3762,"num_write_kb":30095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2213}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3926","reported_seq":3999,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.924870+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:28.924870+0000","last_peered":"2024-09-11T04:57:28.924870+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:28.924870+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32152144,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3867","reported_seq":3916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.843528+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:26.843528+0000","last_peered":"2024-09-11T04:57:26.843528+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:26.843528+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642784,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1576}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3905","reported_seq":3983,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.945479+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:28.945479+0000","last_peered":"2024-09-11T04:57:28.945479+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:28.945479+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31962272,"num_objects":3902,"num_object_clones":0,"num_object_copies":15608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":31221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1584}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3979","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.179215+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:31.179215+0000","last_peered":"2024-09-11T04:57:31.179215+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:31.179215+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3979,"log_dups_size":0,"ondisk_log_size":3979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32552096,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":31804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":508027072,"num_objects":62020,"num_object_clones":0,"num_object_copies":248080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62047,"num_write_kb":496258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":866336768,"data_stored":866336768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62074,"ondisk_log_size":62074,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460776,"kb_used_data":244440,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93911064,"statfs":{"total":96636764160,"available":96164929536,"internally_reserved":0,"allocated":250306560,"data_stored":250135750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,5,1,4,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459208,"kb_used_data":247540,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93912632,"statfs":{"total":96636764160,"available":96166535168,"internally_reserved":0,"allocated":253480960,"data_stored":253310150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221896,"kb_used_data":100024,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94149944,"statfs":{"total":96636764160,"available":96409542656,"internally_reserved":0,"allocated":102424576,"data_stored":102253766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124776146},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467756,"kb_used_data":251484,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93904084,"statfs":{"total":96636764160,"available":96157782016,"internally_reserved":0,"allocated":257519616,"data_stored":257348806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257458176,"data_stored":257458176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103415808,"data_stored":103415808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254115840,"data_stored":254115840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251346944,"data_stored":251346944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:32.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:32.610 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:32.610 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:32.646 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-11T04:57:31.641969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508027072,"num_objects":62020,"num_object_clones":0,"num_object_copies":248080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62047,"num_write_kb":496258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62074,"ondisk_log_size":62074,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1609636,"kb_used_data":843488,"kb_used_omap":14,"kb_used_meta":766065,"kb_avail":375877724,"statfs":{"total":386547056640,"available":384898789376,"internally_reserved":0,"allocated":863731712,"data_stored":863048472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":784451126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,14,12,10,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32841728,"num_objects":4009,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":32072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005246"},"pg_stats":[{"pgid":"1.f","version":"31'3744","reported_seq":3785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.852705+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:26.852705+0000","last_peered":"2024-09-11T04:57:26.852705+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:26.852705+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3744,"log_dups_size":0,"ondisk_log_size":3744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3938","reported_seq":3999,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.120410+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:31.120410+0000","last_peered":"2024-09-11T04:57:31.120410+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:31.120410+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32224416,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3719","reported_seq":3792,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.939080+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:28.939080+0000","last_peered":"2024-09-11T04:57:28.939080+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:28.939080+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30440016,"num_objects":3716,"num_object_clones":0,"num_object_copies":14864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":29734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3867","reported_seq":3930,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.088228+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:31.088228+0000","last_peered":"2024-09-11T04:57:31.088228+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:31.088228+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642784,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3855","reported_seq":3908,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.742150+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:26.742150+0000","last_peered":"2024-09-11T04:57:26.742150+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:26.742150+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31552672,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":30821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3855","reported_seq":3895,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.837984+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:26.837984+0000","last_peered":"2024-09-11T04:57:26.837984+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:26.837984+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31580160,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":30840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1534}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3957","reported_seq":4019,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.181862+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:31.181862+0000","last_peered":"2024-09-11T04:57:31.181862+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:31.181862+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363680,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3955,"num_write_kb":31620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1634}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4046","reported_seq":4108,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.271765+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:31.271765+0000","last_peered":"2024-09-11T04:57:31.271765+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:31.271765+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":32340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3924","reported_seq":3979,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.777291+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:26.777291+0000","last_peered":"2024-09-11T04:57:26.777291+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:26.777291+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32074048,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3921,"num_write_kb":31345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3884","reported_seq":3960,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.899015+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:28.899015+0000","last_peered":"2024-09-11T04:57:28.899015+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:28.899015+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31798432,"num_objects":3882,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1564}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3845","reported_seq":3900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.805160+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:26.805160+0000","last_peered":"2024-09-11T04:57:26.805160+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:26.805160+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31452912,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":30724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3763","reported_seq":5171,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.855733+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:26.855733+0000","last_peered":"2024-09-11T04:57:26.855733+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:26.855733+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3763,"log_dups_size":0,"ondisk_log_size":3763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30816848,"num_objects":3762,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3762,"num_write_kb":30095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2213}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3926","reported_seq":3999,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.924870+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:28.924870+0000","last_peered":"2024-09-11T04:57:28.924870+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:28.924870+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32152144,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3867","reported_seq":3916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:26.843528+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:26.843528+0000","last_peered":"2024-09-11T04:57:26.843528+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:26.843528+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642784,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1576}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3905","reported_seq":3983,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.945479+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:28.945479+0000","last_peered":"2024-09-11T04:57:28.945479+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:28.945479+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31962272,"num_objects":3902,"num_object_clones":0,"num_object_copies":15608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":31221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1584}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3979","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.179215+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:31.179215+0000","last_peered":"2024-09-11T04:57:31.179215+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:31.179215+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3979,"log_dups_size":0,"ondisk_log_size":3979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32552096,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":31804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":508027072,"num_objects":62020,"num_object_clones":0,"num_object_copies":248080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62047,"num_write_kb":496258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":866336768,"data_stored":866336768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62074,"ondisk_log_size":62074,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460776,"kb_used_data":244440,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93911064,"statfs":{"total":96636764160,"available":96164929536,"internally_reserved":0,"allocated":250306560,"data_stored":250135750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,5,1,4,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459208,"kb_used_data":247540,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93912632,"statfs":{"total":96636764160,"available":96166535168,"internally_reserved":0,"allocated":253480960,"data_stored":253310150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221896,"kb_used_data":100024,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94149944,"statfs":{"total":96636764160,"available":96409542656,"internally_reserved":0,"allocated":102424576,"data_stored":102253766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":124776146},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467756,"kb_used_data":251484,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93904084,"statfs":{"total":96636764160,"available":96157782016,"internally_reserved":0,"allocated":257519616,"data_stored":257348806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257458176,"data_stored":257458176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103415808,"data_stored":103415808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254115840,"data_stored":254115840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251346944,"data_stored":251346944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:32.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:32.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3744", 'reported_seq': 3785, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:26.852705+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:26.852705+0000', 'last_peered': '2024-09-11T04:57:26.852705+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:26.852705+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3744, 'log_dups_size': 0, 'ondisk_log_size': 3744, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30670848, 'num_objects': 3744, 'num_object_clones': 0, 'num_object_copies': 14976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3744, 'num_write_kb': 29952, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1432}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:32.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:32.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3938", 'reported_seq': 3999, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:31.120410+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:31.120410+0000', 'last_peered': '2024-09-11T04:57:31.120410+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:31.120410+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3938, 'log_dups_size': 0, 'ondisk_log_size': 3938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32224416, 'num_objects': 3934, 'num_object_clones': 0, 'num_object_copies': 15736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3936, 'num_write_kb': 31484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1655}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:32.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:32.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3719", 'reported_seq': 3792, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:28.939080+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:28.939080+0000', 'last_peered': '2024-09-11T04:57:28.939080+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:28.939080+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3719, 'log_dups_size': 0, 'ondisk_log_size': 3719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30440016, 'num_objects': 3716, 'num_object_clones': 0, 'num_object_copies': 14864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3718, 'num_write_kb': 29734, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1496}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:32.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:32.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3867", 'reported_seq': 3930, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:31.088228+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:31.088228+0000', 'last_peered': '2024-09-11T04:57:31.088228+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:31.088228+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3867, 'log_dups_size': 0, 'ondisk_log_size': 3867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31642784, 'num_objects': 3863, 'num_object_clones': 0, 'num_object_copies': 15452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3865, 'num_write_kb': 30909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1585}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3855", 'reported_seq': 3908, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:26.742150+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:26.742150+0000', 'last_peered': '2024-09-11T04:57:26.742150+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:26.742150+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3855, 'log_dups_size': 0, 'ondisk_log_size': 3855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31552672, 'num_objects': 3852, 'num_object_clones': 0, 'num_object_copies': 15408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3853, 'num_write_kb': 30821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1602}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3855", 'reported_seq': 3895, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:26.837984+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:26.837984+0000', 'last_peered': '2024-09-11T04:57:26.837984+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:26.837984+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3855, 'log_dups_size': 0, 'ondisk_log_size': 3855, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31580160, 'num_objects': 3855, 'num_object_clones': 0, 'num_object_copies': 15420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3855, 'num_write_kb': 30840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1534}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3957", 'reported_seq': 4019, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:31.181862+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:31.181862+0000', 'last_peered': '2024-09-11T04:57:31.181862+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:31.181862+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3957, 'log_dups_size': 0, 'ondisk_log_size': 3957, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32363680, 'num_objects': 3951, 'num_object_clones': 0, 'num_object_copies': 15804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3955, 'num_write_kb': 31620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1634}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4046", 'reported_seq': 4108, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:31.271765+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:31.271765+0000', 'last_peered': '2024-09-11T04:57:31.271765+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:31.271765+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4046, 'log_dups_size': 0, 'ondisk_log_size': 4046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33100960, 'num_objects': 4041, 'num_object_clones': 0, 'num_object_copies': 16164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4044, 'num_write_kb': 32340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1680}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3924", 'reported_seq': 3979, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:26.777291+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:26.777291+0000', 'last_peered': '2024-09-11T04:57:26.777291+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:26.777291+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3924, 'log_dups_size': 0, 'ondisk_log_size': 3924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32074048, 'num_objects': 3916, 'num_object_clones': 0, 'num_object_copies': 15664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3921, 'num_write_kb': 31345, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1522}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:32.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3884", 'reported_seq': 3960, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:28.899015+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:28.899015+0000', 'last_peered': '2024-09-11T04:57:28.899015+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:28.899015+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3884, 'log_dups_size': 0, 'ondisk_log_size': 3884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31798432, 'num_objects': 3882, 'num_object_clones': 0, 'num_object_copies': 15528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3882, 'num_write_kb': 31054, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1564}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3845", 'reported_seq': 3900, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:26.805160+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:26.805160+0000', 'last_peered': '2024-09-11T04:57:26.805160+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:26.805160+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3845, 'log_dups_size': 0, 'ondisk_log_size': 3845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31452912, 'num_objects': 3840, 'num_object_clones': 0, 'num_object_copies': 15360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3842, 'num_write_kb': 30724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1517}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3763", 'reported_seq': 5171, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:26.855733+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:26.855733+0000', 'last_peered': '2024-09-11T04:57:26.855733+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:26.855733+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3763, 'log_dups_size': 0, 'ondisk_log_size': 3763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30816848, 'num_objects': 3762, 'num_object_clones': 0, 'num_object_copies': 15048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1549, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3762, 'num_write_kb': 30095, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 687, 'num_bytes_recovered': 5619712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2213}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3926", 'reported_seq': 3999, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:28.924870+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:28.924870+0000', 'last_peered': '2024-09-11T04:57:28.924870+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:28.924870+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3926, 'log_dups_size': 0, 'ondisk_log_size': 3926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32152144, 'num_objects': 3925, 'num_object_clones': 0, 'num_object_copies': 15700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3925, 'num_write_kb': 31399, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3867", 'reported_seq': 3916, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:26.843528+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:26.843528+0000', 'last_peered': '2024-09-11T04:57:26.843528+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:26.843528+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3867, 'log_dups_size': 0, 'ondisk_log_size': 3867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31642784, 'num_objects': 3863, 'num_object_clones': 0, 'num_object_copies': 15452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3865, 'num_write_kb': 30916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1576}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:32.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3905", 'reported_seq': 3983, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:28.945479+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:28.945479+0000', 'last_peered': '2024-09-11T04:57:28.945479+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:28.945479+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3905, 'log_dups_size': 0, 'ondisk_log_size': 3905, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31962272, 'num_objects': 3902, 'num_object_clones': 0, 'num_object_copies': 15608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3903, 'num_write_kb': 31221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1584}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:32.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:32.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3979", 'reported_seq': 4041, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:31.179215+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:31.179215+0000', 'last_peered': '2024-09-11T04:57:31.179215+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:31.179215+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3979, 'log_dups_size': 0, 'ondisk_log_size': 3979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32552096, 'num_objects': 3974, 'num_object_clones': 0, 'num_object_copies': 15896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3977, 'num_write_kb': 31804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:32.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:32.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 109 20 63303 63283 4.535 2.39844 0.0981004 0.0344371 2024-09-11T04:57:33.440 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:33.440 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36196/248080 objects degraded (14.590%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":180,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62020,"data_bytes":508027072,"bytes_used":1648267264,"bytes_avail":384898789376,"bytes_total":386547056640,"degraded_objects":36196,"degraded_total":248080,"degraded_ratio":0.14590454692034827,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":78472,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"write_bytes_sec":2735614,"read_op_per_sec":0,"write_op_per_sec":333},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:57:33.463 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:33.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:33.701 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 110 20 63615 63595 4.51593 2.4375 0.150219 0.0345837 2024-09-11T04:57:34.137 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:34.138 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:34.169 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-11T04:57:33.642711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513933504,"num_objects":62741,"num_object_clones":0,"num_object_copies":250964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62768,"num_write_kb":502026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":770,"num_bytes_recovered":6086656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62795,"ondisk_log_size":62795,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1635980,"kb_used_data":857352,"kb_used_omap":14,"kb_used_meta":778545,"kb_avail":375851380,"statfs":{"total":386547056640,"available":384871813120,"internally_reserved":0,"allocated":877928448,"data_stored":877245208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":797230646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,13,8,8,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34054144,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":33256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005183"},"pg_stats":[{"pgid":"1.f","version":"31'3864","reported_seq":3905,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.737840+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:31.737840+0000","last_peered":"2024-09-11T04:57:31.737840+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:31.737840+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31653888,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3864,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1552}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3938","reported_seq":3999,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.120410+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:31.120410+0000","last_peered":"2024-09-11T04:57:31.120410+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:31.120410+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32224416,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3719","reported_seq":3792,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.939080+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:28.939080+0000","last_peered":"2024-09-11T04:57:28.939080+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:28.939080+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30440016,"num_objects":3716,"num_object_clones":0,"num_object_copies":14864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":29734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3867","reported_seq":3930,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.088228+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:31.088228+0000","last_peered":"2024-09-11T04:57:31.088228+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:31.088228+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642784,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3946","reported_seq":4000,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.944792+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:31.944792+0000","last_peered":"2024-09-11T04:57:31.944792+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:31.944792+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3946,"log_dups_size":0,"ondisk_log_size":3946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32298144,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3944,"num_write_kb":31549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3966","reported_seq":4006,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.914412+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:31.914412+0000","last_peered":"2024-09-11T04:57:31.914412+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:31.914412+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32489472,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3966,"num_write_kb":31728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3957","reported_seq":4019,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.181862+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:31.181862+0000","last_peered":"2024-09-11T04:57:31.181862+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:31.181862+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32363680,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3955,"num_write_kb":31620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1634}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4046","reported_seq":4108,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.271765+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:31.271765+0000","last_peered":"2024-09-11T04:57:31.271765+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:31.271765+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33100960,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":32340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1680}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4020","reported_seq":4075,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.817145+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:31.817145+0000","last_peered":"2024-09-11T04:57:31.817145+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:31.817145+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4020,"log_dups_size":0,"ondisk_log_size":4020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32860480,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4017,"num_write_kb":32113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3884","reported_seq":3960,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.899015+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:28.899015+0000","last_peered":"2024-09-11T04:57:28.899015+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:28.899015+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31798432,"num_objects":3882,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1564}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3964","reported_seq":4019,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.934158+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:31.934158+0000","last_peered":"2024-09-11T04:57:31.934158+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:31.934158+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3964,"log_dups_size":0,"ondisk_log_size":3964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32427760,"num_objects":3959,"num_object_clones":0,"num_object_copies":15836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3961,"num_write_kb":31676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1636}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3849","reported_seq":5370,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.901429+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:31.901429+0000","last_peered":"2024-09-11T04:57:31.901429+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:31.901429+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3849,"log_dups_size":0,"ondisk_log_size":3849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31521360,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":6086656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2356}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3926","reported_seq":3999,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.924870+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:28.924870+0000","last_peered":"2024-09-11T04:57:28.924870+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:28.924870+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32152144,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3965","reported_seq":4014,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.762838+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:31.762838+0000","last_peered":"2024-09-11T04:57:31.762838+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:31.762838+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3965,"log_dups_size":0,"ondisk_log_size":3965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32445600,"num_objects":3961,"num_object_clones":0,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3963,"num_write_kb":31700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1674}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3905","reported_seq":3983,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:28.945479+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:28.945479+0000","last_peered":"2024-09-11T04:57:28.945479+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:28.945479+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31962272,"num_objects":3902,"num_object_clones":0,"num_object_copies":15608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":31221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1584}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3979","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:31.179215+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:31.179215+0000","last_peered":"2024-09-11T04:57:31.179215+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:31.179215+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3979,"log_dups_size":0,"ondisk_log_size":3979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32552096,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":31804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":513933504,"num_objects":62741,"num_object_clones":0,"num_object_copies":250964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62768,"num_write_kb":502026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":770,"num_bytes_recovered":6086656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":880578560,"data_stored":880578560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62795,"ondisk_log_size":62795,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471788,"kb_used_data":251292,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93900052,"statfs":{"total":96636764160,"available":96153653248,"internally_reserved":0,"allocated":257323008,"data_stored":257152198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459208,"kb_used_data":247540,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93912632,"statfs":{"total":96636764160,"available":96166535168,"internally_reserved":0,"allocated":253480960,"data_stored":253310150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237228,"kb_used_data":107036,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94134612,"statfs":{"total":96636764160,"available":96393842688,"internally_reserved":0,"allocated":109604864,"data_stored":109434054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":133295826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467756,"kb_used_data":251484,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93904084,"statfs":{"total":96636764160,"available":96157782016,"internally_reserved":0,"allocated":257519616,"data_stored":257348806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257458176,"data_stored":257458176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110645248,"data_stored":110645248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254115840,"data_stored":254115840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258359296,"data_stored":258359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:34.701 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 111 20 63938 63918 4.49798 2.52344 0.0757995 0.0347241 2024-09-11T04:57:35.701 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 112 20 64284 64264 4.48195 2.70312 0.0366878 0.0348477 2024-09-11T04:57:36.701 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 113 20 64633 64613 4.46642 2.72656 0.0410809 0.0349677 2024-09-11T04:57:37.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:37.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 114 20 64969 64949 4.45026 2.625 0.0759524 0.035101 2024-09-11T04:57:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:37.850 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-11T04:57:37.644514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":527605952,"num_objects":64410,"num_object_clones":0,"num_object_copies":257640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64437,"num_write_kb":515378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64464,"ondisk_log_size":64464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1651120,"kb_used_data":884976,"kb_used_omap":14,"kb_used_meta":766065,"kb_avail":375836240,"statfs":{"total":386547056640,"available":384856309760,"internally_reserved":0,"allocated":906215424,"data_stored":905532184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":784451126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,1,3,7,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38043648,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4644,"num_write_kb":37152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004997"},"pg_stats":[{"pgid":"1.f","version":"31'3985","reported_seq":4025,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.981100+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:36.981100+0000","last_peered":"2024-09-11T04:57:36.981100+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:36.981100+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32645120,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":31880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4027","reported_seq":4088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.137769+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:36.137769+0000","last_peered":"2024-09-11T04:57:36.137769+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:36.137769+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32953504,"num_objects":4023,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":32196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3822","reported_seq":3897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:33.923684+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:33.923684+0000","last_peered":"2024-09-11T04:57:33.923684+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:33.923684+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31283792,"num_objects":3819,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":30558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1599}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3970","reported_seq":4033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.135888+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:36.135888+0000","last_peered":"2024-09-11T04:57:36.135888+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:36.135888+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32486560,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1688}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4050","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.941844+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:36.941844+0000","last_peered":"2024-09-11T04:57:36.941844+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:36.941844+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33150112,"num_objects":4047,"num_object_clones":0,"num_object_copies":16188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4048,"num_write_kb":32381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4087","reported_seq":4127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.877149+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:36.877149+0000","last_peered":"2024-09-11T04:57:36.877149+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:36.877149+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4044","reported_seq":4107,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.148459+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:36.148459+0000","last_peered":"2024-09-11T04:57:36.148459+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:36.148459+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076384,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4042,"num_write_kb":32316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4150","reported_seq":4212,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.169986+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:36.169986+0000","last_peered":"2024-09-11T04:57:36.169986+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:36.169986+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33952928,"num_objects":4145,"num_object_clones":0,"num_object_copies":16580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":33172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4121","reported_seq":4176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.940928+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:36.940928+0000","last_peered":"2024-09-11T04:57:36.940928+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:36.940928+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33687872,"num_objects":4113,"num_object_clones":0,"num_object_copies":16452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4118,"num_write_kb":32921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4001","reported_seq":4078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:33.909951+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:33.909951+0000","last_peered":"2024-09-11T04:57:33.909951+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:33.909951+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756896,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3999,"num_write_kb":31990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4075","reported_seq":4131,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.885459+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:36.885459+0000","last_peered":"2024-09-11T04:57:36.885459+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:36.885459+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33337072,"num_objects":4070,"num_object_clones":0,"num_object_copies":16280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4072,"num_write_kb":32564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3924","reported_seq":5560,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.925568+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:36.925568+0000","last_peered":"2024-09-11T04:57:36.925568+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:36.925568+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32135760,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":31383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4047","reported_seq":4121,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:33.880773+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:33.880773+0000","last_peered":"2024-09-11T04:57:33.880773+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:33.880773+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33143376,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4046,"num_write_kb":32367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1714}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4060","reported_seq":4109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.886877+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:36.886877+0000","last_peered":"2024-09-11T04:57:36.886877+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:36.886877+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223840,"num_objects":4056,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4058,"num_write_kb":32460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1769}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4024","reported_seq":4102,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:33.955100+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:33.955100+0000","last_peered":"2024-09-11T04:57:33.955100+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:33.955100+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32937120,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":32173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4077","reported_seq":4139,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.173834+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:36.173834+0000","last_peered":"2024-09-11T04:57:36.173834+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:36.173834+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4075,"num_write_kb":32588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":527605952,"num_objects":64410,"num_object_clones":0,"num_object_copies":257640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64437,"num_write_kb":515378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":907878400,"data_stored":907878400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64464,"ondisk_log_size":64464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471332,"kb_used_data":259156,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93900508,"statfs":{"total":96636764160,"available":96154120192,"internally_reserved":0,"allocated":265375744,"data_stored":265204934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,1,1,2,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470044,"kb_used_data":254220,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93901796,"statfs":{"total":96636764160,"available":96155439104,"internally_reserved":0,"allocated":260321280,"data_stored":260150470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231476,"kb_used_data":113764,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94140364,"statfs":{"total":96636764160,"available":96399732736,"internally_reserved":0,"allocated":116494336,"data_stored":116323526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120516306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478268,"kb_used_data":257836,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93893572,"statfs":{"total":96636764160,"available":96147017728,"internally_reserved":0,"allocated":264024064,"data_stored":263853254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225701583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":264122368,"data_stored":264122368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117694464,"data_stored":117694464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260902912,"data_stored":260902912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265158656,"data_stored":265158656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:37.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:38.468 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:38.468 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:38.506 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-11T04:57:37.644514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":527605952,"num_objects":64410,"num_object_clones":0,"num_object_copies":257640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64437,"num_write_kb":515378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64464,"ondisk_log_size":64464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1651120,"kb_used_data":884976,"kb_used_omap":14,"kb_used_meta":766065,"kb_avail":375836240,"statfs":{"total":386547056640,"available":384856309760,"internally_reserved":0,"allocated":906215424,"data_stored":905532184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":784451126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,1,3,7,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38043648,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4644,"num_write_kb":37152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004997"},"pg_stats":[{"pgid":"1.f","version":"31'3985","reported_seq":4025,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.981100+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:36.981100+0000","last_peered":"2024-09-11T04:57:36.981100+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:36.981100+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32645120,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":31880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4027","reported_seq":4088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.137769+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:36.137769+0000","last_peered":"2024-09-11T04:57:36.137769+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:36.137769+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32953504,"num_objects":4023,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":32196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3822","reported_seq":3897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:33.923684+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:33.923684+0000","last_peered":"2024-09-11T04:57:33.923684+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:33.923684+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31283792,"num_objects":3819,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":30558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1599}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3970","reported_seq":4033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.135888+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:36.135888+0000","last_peered":"2024-09-11T04:57:36.135888+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:36.135888+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32486560,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1688}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4050","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.941844+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:36.941844+0000","last_peered":"2024-09-11T04:57:36.941844+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:36.941844+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33150112,"num_objects":4047,"num_object_clones":0,"num_object_copies":16188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4048,"num_write_kb":32381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4087","reported_seq":4127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.877149+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:36.877149+0000","last_peered":"2024-09-11T04:57:36.877149+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:36.877149+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4044","reported_seq":4107,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.148459+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:36.148459+0000","last_peered":"2024-09-11T04:57:36.148459+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:36.148459+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076384,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4042,"num_write_kb":32316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4150","reported_seq":4212,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.169986+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:36.169986+0000","last_peered":"2024-09-11T04:57:36.169986+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:36.169986+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33952928,"num_objects":4145,"num_object_clones":0,"num_object_copies":16580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":33172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4121","reported_seq":4176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.940928+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:36.940928+0000","last_peered":"2024-09-11T04:57:36.940928+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:36.940928+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33687872,"num_objects":4113,"num_object_clones":0,"num_object_copies":16452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4118,"num_write_kb":32921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4001","reported_seq":4078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:33.909951+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:33.909951+0000","last_peered":"2024-09-11T04:57:33.909951+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:33.909951+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756896,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3999,"num_write_kb":31990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4075","reported_seq":4131,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.885459+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:36.885459+0000","last_peered":"2024-09-11T04:57:36.885459+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:36.885459+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33337072,"num_objects":4070,"num_object_clones":0,"num_object_copies":16280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4072,"num_write_kb":32564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3924","reported_seq":5560,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.925568+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:36.925568+0000","last_peered":"2024-09-11T04:57:36.925568+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:36.925568+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32135760,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":31383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4047","reported_seq":4121,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:33.880773+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:33.880773+0000","last_peered":"2024-09-11T04:57:33.880773+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:33.880773+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33143376,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4046,"num_write_kb":32367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1714}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4060","reported_seq":4109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.886877+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:36.886877+0000","last_peered":"2024-09-11T04:57:36.886877+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:36.886877+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223840,"num_objects":4056,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4058,"num_write_kb":32460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1769}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4024","reported_seq":4102,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:33.955100+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:33.955100+0000","last_peered":"2024-09-11T04:57:33.955100+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:33.955100+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32937120,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":32173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4077","reported_seq":4139,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.173834+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:36.173834+0000","last_peered":"2024-09-11T04:57:36.173834+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:36.173834+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4075,"num_write_kb":32588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":527605952,"num_objects":64410,"num_object_clones":0,"num_object_copies":257640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64437,"num_write_kb":515378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":907878400,"data_stored":907878400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64464,"ondisk_log_size":64464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471332,"kb_used_data":259156,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93900508,"statfs":{"total":96636764160,"available":96154120192,"internally_reserved":0,"allocated":265375744,"data_stored":265204934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,1,1,2,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470044,"kb_used_data":254220,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93901796,"statfs":{"total":96636764160,"available":96155439104,"internally_reserved":0,"allocated":260321280,"data_stored":260150470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231476,"kb_used_data":113764,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94140364,"statfs":{"total":96636764160,"available":96399732736,"internally_reserved":0,"allocated":116494336,"data_stored":116323526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120516306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478268,"kb_used_data":257836,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93893572,"statfs":{"total":96636764160,"available":96147017728,"internally_reserved":0,"allocated":264024064,"data_stored":263853254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225701583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":264122368,"data_stored":264122368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117694464,"data_stored":117694464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260902912,"data_stored":260902912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265158656,"data_stored":265158656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3985", 'reported_seq': 4025, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.981100+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:36.981100+0000', 'last_peered': '2024-09-11T04:57:36.981100+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:36.981100+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32645120, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 15940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 31880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1673}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4027", 'reported_seq': 4088, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.137769+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:36.137769+0000', 'last_peered': '2024-09-11T04:57:36.137769+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:36.137769+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4027, 'log_dups_size': 0, 'ondisk_log_size': 4027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32953504, 'num_objects': 4023, 'num_object_clones': 0, 'num_object_copies': 16092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4023, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4025, 'num_write_kb': 32196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1744}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3822", 'reported_seq': 3897, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:33.923684+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:33.923684+0000', 'last_peered': '2024-09-11T04:57:33.923684+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:33.923684+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3822, 'log_dups_size': 0, 'ondisk_log_size': 3822, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31283792, 'num_objects': 3819, 'num_object_clones': 0, 'num_object_copies': 15276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3821, 'num_write_kb': 30558, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1599}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3970", 'reported_seq': 4033, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.135888+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:36.135888+0000', 'last_peered': '2024-09-11T04:57:36.135888+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:36.135888+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3970, 'log_dups_size': 0, 'ondisk_log_size': 3970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32486560, 'num_objects': 3966, 'num_object_clones': 0, 'num_object_copies': 15864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3968, 'num_write_kb': 31733, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1688}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4050", 'reported_seq': 4104, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.941844+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:36.941844+0000', 'last_peered': '2024-09-11T04:57:36.941844+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:36.941844+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4050, 'log_dups_size': 0, 'ondisk_log_size': 4050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33150112, 'num_objects': 4047, 'num_object_clones': 0, 'num_object_copies': 16188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4048, 'num_write_kb': 32381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1797}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4087", 'reported_seq': 4127, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.877149+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:36.877149+0000', 'last_peered': '2024-09-11T04:57:36.877149+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:36.877149+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4087, 'log_dups_size': 0, 'ondisk_log_size': 4087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33480704, 'num_objects': 4087, 'num_object_clones': 0, 'num_object_copies': 16348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4087, 'num_write_kb': 32696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1766}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4044", 'reported_seq': 4107, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.148459+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:36.148459+0000', 'last_peered': '2024-09-11T04:57:36.148459+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:36.148459+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4044, 'log_dups_size': 0, 'ondisk_log_size': 4044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33076384, 'num_objects': 4038, 'num_object_clones': 0, 'num_object_copies': 16152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4042, 'num_write_kb': 32316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1721}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4150", 'reported_seq': 4212, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.169986+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:36.169986+0000', 'last_peered': '2024-09-11T04:57:36.169986+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:36.169986+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4150, 'log_dups_size': 0, 'ondisk_log_size': 4150, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33952928, 'num_objects': 4145, 'num_object_clones': 0, 'num_object_copies': 16580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4148, 'num_write_kb': 33172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1784}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:38.508 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4121", 'reported_seq': 4176, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.940928+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:36.940928+0000', 'last_peered': '2024-09-11T04:57:36.940928+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:36.940928+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4121, 'log_dups_size': 0, 'ondisk_log_size': 4121, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33687872, 'num_objects': 4113, 'num_object_clones': 0, 'num_object_copies': 16452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4118, 'num_write_kb': 32921, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1719}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4001", 'reported_seq': 4078, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:33.909951+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:33.909951+0000', 'last_peered': '2024-09-11T04:57:33.909951+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:33.909951+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4001, 'log_dups_size': 0, 'ondisk_log_size': 4001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32756896, 'num_objects': 3999, 'num_object_clones': 0, 'num_object_copies': 15996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3999, 'num_write_kb': 31990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1681}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4075", 'reported_seq': 4131, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.885459+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:36.885459+0000', 'last_peered': '2024-09-11T04:57:36.885459+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:36.885459+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4075, 'log_dups_size': 0, 'ondisk_log_size': 4075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33337072, 'num_objects': 4070, 'num_object_clones': 0, 'num_object_copies': 16280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4072, 'num_write_kb': 32564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1747}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3924", 'reported_seq': 5560, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.925568+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:36.925568+0000', 'last_peered': '2024-09-11T04:57:36.925568+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:36.925568+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3924, 'log_dups_size': 0, 'ondisk_log_size': 3924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32135760, 'num_objects': 3923, 'num_object_clones': 0, 'num_object_copies': 15692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1435, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3923, 'num_write_kb': 31383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 801, 'num_bytes_recovered': 6553600, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4047", 'reported_seq': 4121, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:33.880773+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:33.880773+0000', 'last_peered': '2024-09-11T04:57:33.880773+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:33.880773+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4047, 'log_dups_size': 0, 'ondisk_log_size': 4047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33143376, 'num_objects': 4046, 'num_object_clones': 0, 'num_object_copies': 16184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4046, 'num_write_kb': 32367, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1714}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:38.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4060", 'reported_seq': 4109, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.886877+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:36.886877+0000', 'last_peered': '2024-09-11T04:57:36.886877+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:36.886877+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33223840, 'num_objects': 4056, 'num_object_clones': 0, 'num_object_copies': 16224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4058, 'num_write_kb': 32460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1769}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:38.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:38.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4024", 'reported_seq': 4102, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:33.955100+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:33.955100+0000', 'last_peered': '2024-09-11T04:57:33.955100+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:33.955100+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32937120, 'num_objects': 4021, 'num_object_clones': 0, 'num_object_copies': 16084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4022, 'num_write_kb': 32173, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:38.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:38.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4077", 'reported_seq': 4139, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:36.173834+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:36.173834+0000', 'last_peered': '2024-09-11T04:57:36.173834+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:36.173834+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4077, 'log_dups_size': 0, 'ondisk_log_size': 4077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33354912, 'num_objects': 4072, 'num_object_clones': 0, 'num_object_copies': 16288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4075, 'num_write_kb': 32588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1761}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:38.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:38.701 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 115 20 65254 65234 4.43093 2.22656 0.0379244 0.035243 2024-09-11T04:57:39.301 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:39.302 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36082/257640 objects degraded (14.005%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":186,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64410,"data_bytes":527605952,"bytes_used":1690746880,"bytes_avail":384856309760,"bytes_total":386547056640,"degraded_objects":36082,"degraded_total":257640,"degraded_ratio":0.14004812917248874,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":116005,"recovering_keys_per_sec":0,"num_objects_recovered":170,"num_bytes_recovered":1392640,"num_keys_recovered":0,"write_bytes_sec":3168984,"read_op_per_sec":0,"write_op_per_sec":386},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:57:39.333 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:39.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:39.701 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 116 20 65549 65529 4.4126 2.30469 0.0546619 0.0353919 2024-09-11T04:57:39.919 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:39.920 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:39.944 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-11T04:57:39.645233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530956480,"num_objects":64819,"num_object_clones":0,"num_object_copies":259276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64846,"num_write_kb":518650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64873,"ondisk_log_size":64873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1661996,"kb_used_data":891692,"kb_used_omap":14,"kb_used_meta":770225,"kb_avail":375825364,"statfs":{"total":386547056640,"available":384845172736,"internally_reserved":0,"allocated":913092608,"data_stored":912409368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":788710966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,4,3,2,6,14,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30842880,"num_objects":3765,"num_object_clones":0,"num_object_copies":15060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":30120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004852"},"pg_stats":[{"pgid":"1.f","version":"31'3985","reported_seq":4025,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.981100+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:36.981100+0000","last_peered":"2024-09-11T04:57:36.981100+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:36.981100+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32645120,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":31880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4027","reported_seq":4088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.137769+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:36.137769+0000","last_peered":"2024-09-11T04:57:36.137769+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:36.137769+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32953504,"num_objects":4023,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":32196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3919","reported_seq":3994,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.657333+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:38.657333+0000","last_peered":"2024-09-11T04:57:38.657333+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:38.657333+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32078416,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":31334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3970","reported_seq":4033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.135888+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:36.135888+0000","last_peered":"2024-09-11T04:57:36.135888+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:36.135888+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32486560,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1688}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4050","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.941844+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:36.941844+0000","last_peered":"2024-09-11T04:57:36.941844+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:36.941844+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33150112,"num_objects":4047,"num_object_clones":0,"num_object_copies":16188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4048,"num_write_kb":32381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4087","reported_seq":4127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.877149+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:36.877149+0000","last_peered":"2024-09-11T04:57:36.877149+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:36.877149+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4044","reported_seq":4107,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.148459+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:36.148459+0000","last_peered":"2024-09-11T04:57:36.148459+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:36.148459+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076384,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4042,"num_write_kb":32316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4150","reported_seq":4212,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.169986+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:36.169986+0000","last_peered":"2024-09-11T04:57:36.169986+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:36.169986+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33952928,"num_objects":4145,"num_object_clones":0,"num_object_copies":16580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":33172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4121","reported_seq":4176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.940928+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:36.940928+0000","last_peered":"2024-09-11T04:57:36.940928+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:36.940928+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33687872,"num_objects":4113,"num_object_clones":0,"num_object_copies":16452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4118,"num_write_kb":32921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4098","reported_seq":4175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.852366+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:38.852366+0000","last_peered":"2024-09-11T04:57:38.852366+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:38.852366+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551520,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1778}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4075","reported_seq":4131,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.885459+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:36.885459+0000","last_peered":"2024-09-11T04:57:36.885459+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:36.885459+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33337072,"num_objects":4070,"num_object_clones":0,"num_object_copies":16280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4072,"num_write_kb":32564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3924","reported_seq":5560,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.925568+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:36.925568+0000","last_peered":"2024-09-11T04:57:36.925568+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:36.925568+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32135760,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":31383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4159","reported_seq":4233,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.849856+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:38.849856+0000","last_peered":"2024-09-11T04:57:38.849856+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:38.849856+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4060","reported_seq":4109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.886877+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:36.886877+0000","last_peered":"2024-09-11T04:57:36.886877+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:36.886877+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223840,"num_objects":4056,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4058,"num_write_kb":32460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1769}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4127","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.966378+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:38.966378+0000","last_peered":"2024-09-11T04:57:38.966378+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:38.966378+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33780896,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4077","reported_seq":4139,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.173834+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:36.173834+0000","last_peered":"2024-09-11T04:57:36.173834+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:36.173834+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4075,"num_write_kb":32588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530956480,"num_objects":64819,"num_object_clones":0,"num_object_copies":259276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64846,"num_write_kb":518650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":914604032,"data_stored":914604032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64873,"ondisk_log_size":64873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471332,"kb_used_data":259156,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93900508,"statfs":{"total":96636764160,"available":96154120192,"internally_reserved":0,"allocated":265375744,"data_stored":265204934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,1,1,2,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480920,"kb_used_data":260936,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93890920,"statfs":{"total":96636764160,"available":96144302080,"internally_reserved":0,"allocated":267198464,"data_stored":267027654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,0,1,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231476,"kb_used_data":113764,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94140364,"statfs":{"total":96636764160,"available":96399732736,"internally_reserved":0,"allocated":116494336,"data_stored":116323526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120516306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478268,"kb_used_data":257836,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93893572,"statfs":{"total":96636764160,"available":96147017728,"internally_reserved":0,"allocated":264024064,"data_stored":263853254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225701583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":264122368,"data_stored":264122368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117694464,"data_stored":117694464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267628544,"data_stored":267628544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265158656,"data_stored":265158656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:40.701 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 117 20 65870 65850 4.3963 2.50781 0.0334573 0.0355289 2024-09-11T04:57:41.701 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 118 20 66214 66194 4.38182 2.6875 0.0570675 0.035648 2024-09-11T04:57:42.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 119 20 66548 66528 4.36693 2.60938 0.0821272 0.0357646 2024-09-11T04:57:42.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:43.605 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:43.606 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:43.639 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-11T04:57:41.645983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":535142592,"num_objects":65330,"num_object_clones":0,"num_object_copies":261320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65357,"num_write_kb":522738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65384,"ondisk_log_size":65384,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1672544,"kb_used_data":898080,"kb_used_omap":14,"kb_used_meta":774385,"kb_avail":375814816,"statfs":{"total":386547056640,"available":384834371584,"internally_reserved":0,"allocated":919633920,"data_stored":918950680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":792970806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,5,3,4,7,15,20,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31596544,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3857,"num_write_kb":30856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004692"},"pg_stats":[{"pgid":"1.f","version":"31'3985","reported_seq":4025,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.981100+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:36.981100+0000","last_peered":"2024-09-11T04:57:36.981100+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:36.981100+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32645120,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":31880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4127","reported_seq":4189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.252198+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:41.252198+0000","last_peered":"2024-09-11T04:57:41.252198+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:41.252198+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772704,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3919","reported_seq":3994,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.657333+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:38.657333+0000","last_peered":"2024-09-11T04:57:38.657333+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:38.657333+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32078416,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":31334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4076","reported_seq":4139,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.303850+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:41.303850+0000","last_peered":"2024-09-11T04:57:41.303850+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:41.303850+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":32581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4050","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.941844+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:36.941844+0000","last_peered":"2024-09-11T04:57:36.941844+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:36.941844+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33150112,"num_objects":4047,"num_object_clones":0,"num_object_copies":16188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4048,"num_write_kb":32381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4087","reported_seq":4127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.877149+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:36.877149+0000","last_peered":"2024-09-11T04:57:36.877149+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:36.877149+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4137","reported_seq":4200,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.228285+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:41.228285+0000","last_peered":"2024-09-11T04:57:41.228285+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:41.228285+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838240,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4254","reported_seq":4315,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.296784+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:41.296784+0000","last_peered":"2024-09-11T04:57:41.296784+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:41.296784+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34804896,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4252,"num_write_kb":34004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4121","reported_seq":4176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.940928+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:36.940928+0000","last_peered":"2024-09-11T04:57:36.940928+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:36.940928+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33687872,"num_objects":4113,"num_object_clones":0,"num_object_copies":16452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4118,"num_write_kb":32921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4098","reported_seq":4175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.852366+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:38.852366+0000","last_peered":"2024-09-11T04:57:38.852366+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:38.852366+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551520,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1778}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4075","reported_seq":4131,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.885459+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:36.885459+0000","last_peered":"2024-09-11T04:57:36.885459+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:36.885459+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33337072,"num_objects":4070,"num_object_clones":0,"num_object_copies":16280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4072,"num_write_kb":32564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3924","reported_seq":5560,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.925568+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:36.925568+0000","last_peered":"2024-09-11T04:57:36.925568+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:36.925568+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32135760,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":31383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4159","reported_seq":4233,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.849856+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:38.849856+0000","last_peered":"2024-09-11T04:57:38.849856+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:38.849856+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4060","reported_seq":4109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:36.886877+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:36.886877+0000","last_peered":"2024-09-11T04:57:36.886877+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:36.886877+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223840,"num_objects":4056,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4058,"num_write_kb":32460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1769}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4127","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.966378+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:38.966378+0000","last_peered":"2024-09-11T04:57:38.966378+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:38.966378+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33780896,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4185","reported_seq":4248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.307285+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:41.307285+0000","last_peered":"2024-09-11T04:57:41.307285+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:41.307285+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34239648,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":33452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":535142592,"num_objects":65330,"num_object_clones":0,"num_object_copies":261320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65357,"num_write_kb":522738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":6553600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":921141248,"data_stored":921141248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65384,"ondisk_log_size":65384,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471332,"kb_used_data":259156,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93900508,"statfs":{"total":96636764160,"available":96154120192,"internally_reserved":0,"allocated":265375744,"data_stored":265204934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,1,1,2,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480920,"kb_used_data":260936,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93890920,"statfs":{"total":96636764160,"available":96144302080,"internally_reserved":0,"allocated":267198464,"data_stored":267027654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,0,1,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231476,"kb_used_data":113764,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94140364,"statfs":{"total":96636764160,"available":96399732736,"internally_reserved":0,"allocated":116494336,"data_stored":116323526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":120516306},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488816,"kb_used_data":264224,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93883024,"statfs":{"total":96636764160,"available":96136216576,"internally_reserved":0,"allocated":270565376,"data_stored":270394566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270659584,"data_stored":270659584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117694464,"data_stored":117694464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267628544,"data_stored":267628544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265158656,"data_stored":265158656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:43.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:43.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:57:43.702928+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0358989 2024-09-11T04:57:43.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:57:43.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 120 20 66857 66837 4.35065 2.41406 0.103326 0.0358989 2024-09-11T04:57:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:44.242 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:44.274 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-11T04:57:43.646931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":540909760,"num_objects":66034,"num_object_clones":0,"num_object_copies":264136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66061,"num_write_kb":528370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":7004160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66088,"ondisk_log_size":66088,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1698132,"kb_used_data":911184,"kb_used_omap":14,"kb_used_meta":786865,"kb_avail":375789228,"statfs":{"total":386547056640,"available":384808169472,"internally_reserved":0,"allocated":933052416,"data_stored":932369176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":805750326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,4,6,8,15,20],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32882688,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4014,"num_write_kb":32112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004961"},"pg_stats":[{"pgid":"1.f","version":"31'4087","reported_seq":4126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:42.051567+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:42.051567+0000","last_peered":"2024-09-11T04:57:42.051567+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:42.051567+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4127","reported_seq":4189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.252198+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:41.252198+0000","last_peered":"2024-09-11T04:57:41.252198+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:41.252198+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772704,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3919","reported_seq":3994,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.657333+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:38.657333+0000","last_peered":"2024-09-11T04:57:38.657333+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:38.657333+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32078416,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":31334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4076","reported_seq":4139,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.303850+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:41.303850+0000","last_peered":"2024-09-11T04:57:41.303850+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:41.303850+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":32581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4135","reported_seq":4188,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.954631+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:41.954631+0000","last_peered":"2024-09-11T04:57:41.954631+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:41.954631+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33846432,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4133,"num_write_kb":33061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4195","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:42.025173+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:42.025173+0000","last_peered":"2024-09-11T04:57:42.025173+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:42.025173+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34365440,"num_objects":4195,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4137","reported_seq":4200,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.228285+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:41.228285+0000","last_peered":"2024-09-11T04:57:41.228285+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:41.228285+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838240,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4254","reported_seq":4315,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.296784+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:41.296784+0000","last_peered":"2024-09-11T04:57:41.296784+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:41.296784+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34804896,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4252,"num_write_kb":34004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4213","reported_seq":4269,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.878553+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:41.878553+0000","last_peered":"2024-09-11T04:57:41.878553+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:41.878553+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34441536,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4098","reported_seq":4175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.852366+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:38.852366+0000","last_peered":"2024-09-11T04:57:38.852366+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:38.852366+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551520,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1778}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4179","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.903667+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:41.903667+0000","last_peered":"2024-09-11T04:57:41.903667+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:41.903667+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34189040,"num_objects":4174,"num_object_clones":0,"num_object_copies":16696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4176,"num_write_kb":33396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4032","reported_seq":5778,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:42.060294+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:42.060294+0000","last_peered":"2024-09-11T04:57:42.060294+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:42.060294+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33020496,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":32247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":7004160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4159","reported_seq":4233,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.849856+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:38.849856+0000","last_peered":"2024-09-11T04:57:38.849856+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:38.849856+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060880,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4165","reported_seq":4214,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.957490+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:41.957490+0000","last_peered":"2024-09-11T04:57:41.957490+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:41.957490+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4165,"log_dups_size":0,"ondisk_log_size":4165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34084000,"num_objects":4161,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4163,"num_write_kb":33300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1874}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4127","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:38.966378+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:38.966378+0000","last_peered":"2024-09-11T04:57:38.966378+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:38.966378+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33780896,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4185","reported_seq":4248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.307285+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:41.307285+0000","last_peered":"2024-09-11T04:57:41.307285+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:41.307285+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34239648,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":33452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":540909760,"num_objects":66034,"num_object_clones":0,"num_object_copies":264136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66061,"num_write_kb":528370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":7004160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":934350848,"data_stored":934350848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66088,"ondisk_log_size":66088,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481880,"kb_used_data":265544,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93889960,"statfs":{"total":96636764160,"available":96143319040,"internally_reserved":0,"allocated":271917056,"data_stored":271746246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,6,4,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480920,"kb_used_data":260936,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93890920,"statfs":{"total":96636764160,"available":96144302080,"internally_reserved":0,"allocated":267198464,"data_stored":267027654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,0,1,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246516,"kb_used_data":120480,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94125324,"statfs":{"total":96636764160,"available":96384331776,"internally_reserved":0,"allocated":123371520,"data_stored":123200710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129035986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488816,"kb_used_data":264224,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93883024,"statfs":{"total":96636764160,"available":96136216576,"internally_reserved":0,"allocated":270565376,"data_stored":270394566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270659584,"data_stored":270659584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124383232,"data_stored":124383232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267628544,"data_stored":267628544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271679488,"data_stored":271679488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:44.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:44.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4087", 'reported_seq': 4126, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:42.051567+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:42.051567+0000', 'last_peered': '2024-09-11T04:57:42.051567+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:42.051567+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4087, 'log_dups_size': 0, 'ondisk_log_size': 4087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33480704, 'num_objects': 4087, 'num_object_clones': 0, 'num_object_copies': 16348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4087, 'num_write_kb': 32696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1775}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:44.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:44.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4127", 'reported_seq': 4189, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.252198+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:41.252198+0000', 'last_peered': '2024-09-11T04:57:41.252198+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:41.252198+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4127, 'log_dups_size': 0, 'ondisk_log_size': 4127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33772704, 'num_objects': 4123, 'num_object_clones': 0, 'num_object_copies': 16492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4125, 'num_write_kb': 32996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:44.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3919", 'reported_seq': 3994, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:38.657333+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:38.657333+0000', 'last_peered': '2024-09-11T04:57:38.657333+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:38.657333+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 3919, 'log_dups_size': 0, 'ondisk_log_size': 3919, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32078416, 'num_objects': 3916, 'num_object_clones': 0, 'num_object_copies': 15664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3918, 'num_write_kb': 31334, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1696}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4076", 'reported_seq': 4139, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.303850+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:41.303850+0000', 'last_peered': '2024-09-11T04:57:41.303850+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:41.303850+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4076, 'log_dups_size': 0, 'ondisk_log_size': 4076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33354912, 'num_objects': 4072, 'num_object_clones': 0, 'num_object_copies': 16288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4074, 'num_write_kb': 32581, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4135", 'reported_seq': 4188, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.954631+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:41.954631+0000', 'last_peered': '2024-09-11T04:57:41.954631+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:41.954631+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4135, 'log_dups_size': 0, 'ondisk_log_size': 4135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33846432, 'num_objects': 4132, 'num_object_clones': 0, 'num_object_copies': 16528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4133, 'num_write_kb': 33061, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1882}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4195", 'reported_seq': 4235, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:42.025173+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:42.025173+0000', 'last_peered': '2024-09-11T04:57:42.025173+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:42.025173+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4195, 'log_dups_size': 0, 'ondisk_log_size': 4195, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34365440, 'num_objects': 4195, 'num_object_clones': 0, 'num_object_copies': 16780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4195, 'num_write_kb': 33560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1874}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4137", 'reported_seq': 4200, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.228285+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:41.228285+0000', 'last_peered': '2024-09-11T04:57:41.228285+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:41.228285+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4137, 'log_dups_size': 0, 'ondisk_log_size': 4137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33838240, 'num_objects': 4131, 'num_object_clones': 0, 'num_object_copies': 16524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4135, 'num_write_kb': 33060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1814}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:44.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4254", 'reported_seq': 4315, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.296784+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:41.296784+0000', 'last_peered': '2024-09-11T04:57:41.296784+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:41.296784+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4254, 'log_dups_size': 0, 'ondisk_log_size': 4254, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34804896, 'num_objects': 4249, 'num_object_clones': 0, 'num_object_copies': 16996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4252, 'num_write_kb': 34004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4213", 'reported_seq': 4269, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.878553+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:41.878553+0000', 'last_peered': '2024-09-11T04:57:41.878553+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:41.878553+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4213, 'log_dups_size': 0, 'ondisk_log_size': 4213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34441536, 'num_objects': 4205, 'num_object_clones': 0, 'num_object_copies': 16820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4210, 'num_write_kb': 33657, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4098", 'reported_seq': 4175, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:38.852366+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:38.852366+0000', 'last_peered': '2024-09-11T04:57:38.852366+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:38.852366+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4098, 'log_dups_size': 0, 'ondisk_log_size': 4098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33551520, 'num_objects': 4096, 'num_object_clones': 0, 'num_object_copies': 16384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4096, 'num_write_kb': 32766, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1778}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4179", 'reported_seq': 4235, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.903667+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:41.903667+0000', 'last_peered': '2024-09-11T04:57:41.903667+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:41.903667+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4179, 'log_dups_size': 0, 'ondisk_log_size': 4179, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34189040, 'num_objects': 4174, 'num_object_clones': 0, 'num_object_copies': 16696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4176, 'num_write_kb': 33396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1851}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4032", 'reported_seq': 5778, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:42.060294+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:42.060294+0000', 'last_peered': '2024-09-11T04:57:42.060294+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:42.060294+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4032, 'log_dups_size': 0, 'ondisk_log_size': 4032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33020496, 'num_objects': 4031, 'num_object_clones': 0, 'num_object_copies': 16124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1380, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4031, 'num_write_kb': 32247, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 856, 'num_bytes_recovered': 7004160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4159", 'reported_seq': 4233, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:38.849856+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:38.849856+0000', 'last_peered': '2024-09-11T04:57:38.849856+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:38.849856+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4159, 'log_dups_size': 0, 'ondisk_log_size': 4159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34060880, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4158, 'num_write_kb': 33263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1826}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:44.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:44.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4165", 'reported_seq': 4214, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.957490+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:41.957490+0000', 'last_peered': '2024-09-11T04:57:41.957490+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:41.957490+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4165, 'log_dups_size': 0, 'ondisk_log_size': 4165, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34084000, 'num_objects': 4161, 'num_object_clones': 0, 'num_object_copies': 16644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4163, 'num_write_kb': 33300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1874}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:44.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:44.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4127", 'reported_seq': 4205, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:38.966378+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:38.966378+0000', 'last_peered': '2024-09-11T04:57:38.966378+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:38.966378+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4127, 'log_dups_size': 0, 'ondisk_log_size': 4127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33780896, 'num_objects': 4124, 'num_object_clones': 0, 'num_object_copies': 16496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4125, 'num_write_kb': 32997, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1806}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:44.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:44.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4185", 'reported_seq': 4248, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:41.307285+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:41.307285+0000', 'last_peered': '2024-09-11T04:57:41.307285+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:41.307285+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4185, 'log_dups_size': 0, 'ondisk_log_size': 4185, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34239648, 'num_objects': 4180, 'num_object_clones': 0, 'num_object_copies': 16720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4183, 'num_write_kb': 33452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:44.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:44.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 121 20 67160 67140 4.33426 2.36719 0.0578756 0.0360368 2024-09-11T04:57:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:45.035 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36027/264136 objects degraded (13.640%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":192,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66034,"data_bytes":540909760,"bytes_used":1738887168,"bytes_avail":384808169472,"bytes_total":386547056640,"degraded_objects":36027,"degraded_total":264136,"degraded_ratio":0.13639564466789836,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":115322,"recovering_keys_per_sec":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"write_bytes_sec":2739091,"read_op_per_sec":0,"write_op_per_sec":334},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:57:45.058 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:45.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:45.660 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:45.696 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-11T04:57:45.647907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543957184,"num_objects":66406,"num_object_clones":0,"num_object_copies":265624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66433,"num_write_kb":531346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":7004160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66460,"ondisk_log_size":66460,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1692052,"kb_used_data":917584,"kb_used_omap":14,"kb_used_meta":774385,"kb_avail":375795308,"statfs":{"total":386547056640,"available":384814395392,"internally_reserved":0,"allocated":939606016,"data_stored":938922776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":792970806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,3,8,7,10,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":29320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005196"},"pg_stats":[{"pgid":"1.f","version":"31'4087","reported_seq":4126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:42.051567+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:42.051567+0000","last_peered":"2024-09-11T04:57:42.051567+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:42.051567+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4127","reported_seq":4189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.252198+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:41.252198+0000","last_peered":"2024-09-11T04:57:41.252198+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:41.252198+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772704,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4011","reported_seq":4086,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:43.947554+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:43.947554+0000","last_peered":"2024-09-11T04:57:43.947554+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:43.947554+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4011,"log_dups_size":0,"ondisk_log_size":4011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32832080,"num_objects":4008,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4010,"num_write_kb":32070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1788}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4076","reported_seq":4139,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.303850+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:41.303850+0000","last_peered":"2024-09-11T04:57:41.303850+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:41.303850+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":32581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4135","reported_seq":4188,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.954631+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:41.954631+0000","last_peered":"2024-09-11T04:57:41.954631+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:41.954631+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33846432,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4133,"num_write_kb":33061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4195","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:42.025173+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:42.025173+0000","last_peered":"2024-09-11T04:57:42.025173+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:42.025173+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34365440,"num_objects":4195,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4137","reported_seq":4200,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.228285+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:41.228285+0000","last_peered":"2024-09-11T04:57:41.228285+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:41.228285+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838240,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1814}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4254","reported_seq":4315,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.296784+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:41.296784+0000","last_peered":"2024-09-11T04:57:41.296784+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:41.296784+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34804896,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4252,"num_write_kb":34004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4213","reported_seq":4269,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.878553+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:41.878553+0000","last_peered":"2024-09-11T04:57:41.878553+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:41.878553+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34441536,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4195","reported_seq":4272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:43.794962+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:43.794962+0000","last_peered":"2024-09-11T04:57:43.794962+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:43.794962+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34346144,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":33542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4179","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.903667+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:41.903667+0000","last_peered":"2024-09-11T04:57:41.903667+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:41.903667+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34189040,"num_objects":4174,"num_object_clones":0,"num_object_copies":16696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4176,"num_write_kb":33396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4032","reported_seq":5778,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:42.060294+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:42.060294+0000","last_peered":"2024-09-11T04:57:42.060294+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:42.060294+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33020496,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":32247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":7004160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4249","reported_seq":4323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:43.887177+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:43.887177+0000","last_peered":"2024-09-11T04:57:43.887177+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:43.887177+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34798160,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":33983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4165","reported_seq":4214,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.957490+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:41.957490+0000","last_peered":"2024-09-11T04:57:41.957490+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:41.957490+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4165,"log_dups_size":0,"ondisk_log_size":4165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34084000,"num_objects":4161,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4163,"num_write_kb":33300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1874}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4220","reported_seq":4298,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:44.050941+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:44.050941+0000","last_peered":"2024-09-11T04:57:44.050941+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:44.050941+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542752,"num_objects":4217,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":33741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1899}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4185","reported_seq":4248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:41.307285+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:41.307285+0000","last_peered":"2024-09-11T04:57:41.307285+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:41.307285+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34239648,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":33452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":543957184,"num_objects":66406,"num_object_clones":0,"num_object_copies":265624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66433,"num_write_kb":531346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":7004160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":941010944,"data_stored":941010944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66460,"ondisk_log_size":66460,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481880,"kb_used_data":265544,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93889960,"statfs":{"total":96636764160,"available":96143319040,"internally_reserved":0,"allocated":271917056,"data_stored":271746246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,6,4,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474840,"kb_used_data":267336,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93897000,"statfs":{"total":96636764160,"available":96150528000,"internally_reserved":0,"allocated":273752064,"data_stored":273581254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246516,"kb_used_data":120480,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94125324,"statfs":{"total":96636764160,"available":96384331776,"internally_reserved":0,"allocated":123371520,"data_stored":123200710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":129035986},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488816,"kb_used_data":264224,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93883024,"statfs":{"total":96636764160,"available":96136216576,"internally_reserved":0,"allocated":270565376,"data_stored":270394566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270659584,"data_stored":270659584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124383232,"data_stored":124383232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274288640,"data_stored":274288640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271679488,"data_stored":271679488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:45.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 122 20 67464 67444 4.3182 2.375 0.0295548 0.0361703 2024-09-11T04:57:46.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 123 20 67805 67785 4.30475 2.66406 0.0581343 0.0362845 2024-09-11T04:57:47.705 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 124 20 68137 68117 4.29095 2.59375 0.0404313 0.0363989 2024-09-11T04:57:48.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:48.704 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 125 20 68477 68457 4.27787 2.65625 0.0471595 0.0365103 2024-09-11T04:57:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:49.305 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:49.329 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-11T04:57:47.648787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":554123456,"num_objects":67647,"num_object_clones":0,"num_object_copies":270588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67674,"num_write_kb":541274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67701,"ondisk_log_size":67701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1725172,"kb_used_data":936820,"kb_used_omap":14,"kb_used_meta":788273,"kb_avail":375762188,"statfs":{"total":386547056640,"available":384780480512,"internally_reserved":0,"allocated":959303680,"data_stored":958620440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":807192118},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,3,4,5,9,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36421632,"num_objects":4446,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4446,"num_write_kb":35568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005136"},"pg_stats":[{"pgid":"1.f","version":"31'4181","reported_seq":4221,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.068879+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:47.068879+0000","last_peered":"2024-09-11T04:57:47.068879+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:47.068879+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34250752,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4181,"num_write_kb":33448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4220","reported_seq":4282,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.201204+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:46.201204+0000","last_peered":"2024-09-11T04:57:46.201204+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:46.201204+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34534560,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":33740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4011","reported_seq":4086,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:43.947554+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:43.947554+0000","last_peered":"2024-09-11T04:57:43.947554+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:43.947554+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4011,"log_dups_size":0,"ondisk_log_size":4011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32832080,"num_objects":4008,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4010,"num_write_kb":32070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1788}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4197","reported_seq":4260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.352087+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:46.352087+0000","last_peered":"2024-09-11T04:57:46.352087+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:46.352087+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34346144,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4242","reported_seq":4296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.072851+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:47.072851+0000","last_peered":"2024-09-11T04:57:47.072851+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:47.072851+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34722976,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4240,"num_write_kb":33917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4288","reported_seq":4326,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.952565+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:46.952565+0000","last_peered":"2024-09-11T04:57:46.952565+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:46.952565+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4247","reported_seq":4309,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.297611+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:46.297611+0000","last_peered":"2024-09-11T04:57:46.297611+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:46.297611+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34739360,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4245,"num_write_kb":33940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1924}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4356","reported_seq":4418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.090492+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:46.090492+0000","last_peered":"2024-09-11T04:57:46.090492+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:46.090492+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4356,"log_dups_size":0,"ondisk_log_size":4356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35640480,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4354,"num_write_kb":34820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4319","reported_seq":4375,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.035986+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:47.035986+0000","last_peered":"2024-09-11T04:57:47.035986+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:47.035986+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4319,"log_dups_size":0,"ondisk_log_size":4319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35309888,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4316,"num_write_kb":34505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4195","reported_seq":4272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:43.794962+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:43.794962+0000","last_peered":"2024-09-11T04:57:43.794962+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:43.794962+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34346144,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":33542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4290","reported_seq":4345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.015814+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:47.015814+0000","last_peered":"2024-09-11T04:57:47.015814+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:47.015814+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35098352,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":34284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1962}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4118","reported_seq":5977,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.053944+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:47.053944+0000","last_peered":"2024-09-11T04:57:47.053944+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:47.053944+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33725008,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":32935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4249","reported_seq":4323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:43.887177+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:43.887177+0000","last_peered":"2024-09-11T04:57:43.887177+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:43.887177+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34798160,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":33983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4280","reported_seq":4329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.015972+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:47.015972+0000","last_peered":"2024-09-11T04:57:47.015972+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:47.015972+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35026080,"num_objects":4276,"num_object_clones":0,"num_object_copies":17104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":34220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4220","reported_seq":4298,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:44.050941+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:44.050941+0000","last_peered":"2024-09-11T04:57:44.050941+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:44.050941+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542752,"num_objects":4217,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":33741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1899}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4288","reported_seq":4351,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.275867+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:46.275867+0000","last_peered":"2024-09-11T04:57:46.275867+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:46.275867+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35083424,"num_objects":4283,"num_object_clones":0,"num_object_copies":17132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4286,"num_write_kb":34276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1972}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":554123456,"num_objects":67647,"num_object_clones":0,"num_object_copies":270588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67674,"num_write_kb":541274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":960978944,"data_stored":960978944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67701,"ondisk_log_size":67701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492304,"kb_used_data":271808,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93879536,"statfs":{"total":96636764160,"available":96132644864,"internally_reserved":0,"allocated":278331392,"data_stored":278160582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474840,"kb_used_data":267336,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93897000,"statfs":{"total":96636764160,"available":96150528000,"internally_reserved":0,"allocated":273752064,"data_stored":273581254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":275224,"kb_used_data":126984,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94096616,"statfs":{"total":96636764160,"available":96354934784,"internally_reserved":0,"allocated":130031616,"data_stored":129860806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":151776978},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482804,"kb_used_data":270692,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93889036,"statfs":{"total":96636764160,"available":96142372864,"internally_reserved":0,"allocated":277188608,"data_stored":277017798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277233664,"data_stored":277233664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131186688,"data_stored":131186688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274288640,"data_stored":274288640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278269952,"data_stored":278269952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:49.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:49.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 126 20 68772 68752 4.26221 2.30469 0.0472448 0.0366398 2024-09-11T04:57:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:49.943 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:49.976 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-11T04:57:49.649599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":557211840,"num_objects":68024,"num_object_clones":0,"num_object_copies":272096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68051,"num_write_kb":544290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68078,"ondisk_log_size":68078,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1735840,"kb_used_data":943328,"kb_used_omap":14,"kb_used_meta":792433,"kb_avail":375751520,"statfs":{"total":386547056640,"available":384769556480,"internally_reserved":0,"allocated":965967872,"data_stored":965284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":811451958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,2,3,13,8,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29605888,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":28912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005085"},"pg_stats":[{"pgid":"1.f","version":"31'4181","reported_seq":4221,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.068879+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:47.068879+0000","last_peered":"2024-09-11T04:57:47.068879+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:47.068879+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34250752,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4181,"num_write_kb":33448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4220","reported_seq":4282,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.201204+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:46.201204+0000","last_peered":"2024-09-11T04:57:46.201204+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:46.201204+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34534560,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":33740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4096","reported_seq":4171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:49.057352+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:49.057352+0000","last_peered":"2024-09-11T04:57:49.057352+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:49.057352+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4096,"log_dups_size":0,"ondisk_log_size":4096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528400,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":32750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4197","reported_seq":4260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.352087+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:46.352087+0000","last_peered":"2024-09-11T04:57:46.352087+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:46.352087+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34346144,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4242","reported_seq":4296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.072851+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:47.072851+0000","last_peered":"2024-09-11T04:57:47.072851+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:47.072851+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34722976,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4240,"num_write_kb":33917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4288","reported_seq":4326,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.952565+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:46.952565+0000","last_peered":"2024-09-11T04:57:46.952565+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:46.952565+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4247","reported_seq":4309,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.297611+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:46.297611+0000","last_peered":"2024-09-11T04:57:46.297611+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:46.297611+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34739360,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4245,"num_write_kb":33940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1924}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4356","reported_seq":4418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.090492+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:46.090492+0000","last_peered":"2024-09-11T04:57:46.090492+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:46.090492+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4356,"log_dups_size":0,"ondisk_log_size":4356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35640480,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4354,"num_write_kb":34820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4319","reported_seq":4375,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.035986+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:47.035986+0000","last_peered":"2024-09-11T04:57:47.035986+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:47.035986+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4319,"log_dups_size":0,"ondisk_log_size":4319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35309888,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4316,"num_write_kb":34505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4286","reported_seq":4363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:48.972235+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:48.972235+0000","last_peered":"2024-09-11T04:57:48.972235+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:48.972235+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35091616,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4284,"num_write_kb":34270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1966}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4290","reported_seq":4345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.015814+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:47.015814+0000","last_peered":"2024-09-11T04:57:47.015814+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:47.015814+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35098352,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":34284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1962}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4118","reported_seq":5977,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.053944+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:47.053944+0000","last_peered":"2024-09-11T04:57:47.053944+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:47.053944+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33725008,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":32935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4350","reported_seq":4424,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:48.957814+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:48.957814+0000","last_peered":"2024-09-11T04:57:48.957814+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:48.957814+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35625552,"num_objects":4349,"num_object_clones":0,"num_object_copies":17396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4349,"num_write_kb":34791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4280","reported_seq":4329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.015972+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:47.015972+0000","last_peered":"2024-09-11T04:57:47.015972+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:47.015972+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35026080,"num_objects":4276,"num_object_clones":0,"num_object_copies":17104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":34220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4320","reported_seq":4398,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:49.078033+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:49.078033+0000","last_peered":"2024-09-11T04:57:49.078033+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:49.078033+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35361952,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4318,"num_write_kb":34541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1999}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4288","reported_seq":4351,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.275867+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:46.275867+0000","last_peered":"2024-09-11T04:57:46.275867+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:46.275867+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35083424,"num_objects":4283,"num_object_clones":0,"num_object_copies":17132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4286,"num_write_kb":34276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1972}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":557211840,"num_objects":68024,"num_object_clones":0,"num_object_copies":272096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68051,"num_write_kb":544290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":967680000,"data_stored":967680000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68078,"ondisk_log_size":68078,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492304,"kb_used_data":271808,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93879536,"statfs":{"total":96636764160,"available":96132644864,"internally_reserved":0,"allocated":278331392,"data_stored":278160582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485508,"kb_used_data":273844,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93886332,"statfs":{"total":96636764160,"available":96139603968,"internally_reserved":0,"allocated":280416256,"data_stored":280245446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,8,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":275224,"kb_used_data":126984,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94096616,"statfs":{"total":96636764160,"available":96354934784,"internally_reserved":0,"allocated":130031616,"data_stored":129860806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":151776978},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482804,"kb_used_data":270692,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93889036,"statfs":{"total":96636764160,"available":96142372864,"internally_reserved":0,"allocated":277188608,"data_stored":277017798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277233664,"data_stored":277233664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131186688,"data_stored":131186688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280989696,"data_stored":280989696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278269952,"data_stored":278269952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:49.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:49.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4181", 'reported_seq': 4221, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:47.068879+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:47.068879+0000', 'last_peered': '2024-09-11T04:57:47.068879+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:47.068879+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4181, 'log_dups_size': 0, 'ondisk_log_size': 4181, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34250752, 'num_objects': 4181, 'num_object_clones': 0, 'num_object_copies': 16724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4181, 'num_write_kb': 33448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1869}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:49.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:49.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4220", 'reported_seq': 4282, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:46.201204+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:46.201204+0000', 'last_peered': '2024-09-11T04:57:46.201204+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:46.201204+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4220, 'log_dups_size': 0, 'ondisk_log_size': 4220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34534560, 'num_objects': 4216, 'num_object_clones': 0, 'num_object_copies': 16864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4218, 'num_write_kb': 33740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:49.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:49.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4096", 'reported_seq': 4171, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:49.057352+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:49.057352+0000', 'last_peered': '2024-09-11T04:57:49.057352+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:49.057352+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4096, 'log_dups_size': 0, 'ondisk_log_size': 4096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33528400, 'num_objects': 4093, 'num_object_clones': 0, 'num_object_copies': 16372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4095, 'num_write_kb': 32750, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1873}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:49.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:49.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4197", 'reported_seq': 4260, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:46.352087+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:46.352087+0000', 'last_peered': '2024-09-11T04:57:46.352087+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:46.352087+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4197, 'log_dups_size': 0, 'ondisk_log_size': 4197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34346144, 'num_objects': 4193, 'num_object_clones': 0, 'num_object_copies': 16772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4195, 'num_write_kb': 33549, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:49.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:49.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4242", 'reported_seq': 4296, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:47.072851+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:47.072851+0000', 'last_peered': '2024-09-11T04:57:47.072851+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:47.072851+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4242, 'log_dups_size': 0, 'ondisk_log_size': 4242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34722976, 'num_objects': 4239, 'num_object_clones': 0, 'num_object_copies': 16956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4240, 'num_write_kb': 33917, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1989}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:49.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:49.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4288", 'reported_seq': 4326, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:46.952565+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:46.952565+0000', 'last_peered': '2024-09-11T04:57:46.952565+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:46.952565+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4288, 'log_dups_size': 0, 'ondisk_log_size': 4288, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35127296, 'num_objects': 4288, 'num_object_clones': 0, 'num_object_copies': 17152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4288, 'num_write_kb': 34304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1967}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4247", 'reported_seq': 4309, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:46.297611+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:46.297611+0000', 'last_peered': '2024-09-11T04:57:46.297611+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:46.297611+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4247, 'log_dups_size': 0, 'ondisk_log_size': 4247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34739360, 'num_objects': 4241, 'num_object_clones': 0, 'num_object_copies': 16964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4245, 'num_write_kb': 33940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1924}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4356", 'reported_seq': 4418, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:46.090492+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:46.090492+0000', 'last_peered': '2024-09-11T04:57:46.090492+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:46.090492+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4356, 'log_dups_size': 0, 'ondisk_log_size': 4356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35640480, 'num_objects': 4351, 'num_object_clones': 0, 'num_object_copies': 17404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4354, 'num_write_kb': 34820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1990}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4319", 'reported_seq': 4375, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:47.035986+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:47.035986+0000', 'last_peered': '2024-09-11T04:57:47.035986+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:47.035986+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4319, 'log_dups_size': 0, 'ondisk_log_size': 4319, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35309888, 'num_objects': 4311, 'num_object_clones': 0, 'num_object_copies': 17244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4316, 'num_write_kb': 34505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1917}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4286", 'reported_seq': 4363, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:48.972235+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:48.972235+0000', 'last_peered': '2024-09-11T04:57:48.972235+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:48.972235+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4286, 'log_dups_size': 0, 'ondisk_log_size': 4286, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35091616, 'num_objects': 4284, 'num_object_clones': 0, 'num_object_copies': 17136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4284, 'num_write_kb': 34270, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1966}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4290", 'reported_seq': 4345, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:47.015814+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:47.015814+0000', 'last_peered': '2024-09-11T04:57:47.015814+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:47.015814+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4290, 'log_dups_size': 0, 'ondisk_log_size': 4290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35098352, 'num_objects': 4285, 'num_object_clones': 0, 'num_object_copies': 17140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4285, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4287, 'num_write_kb': 34284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1962}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:49.979 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4118", 'reported_seq': 5977, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:47.053944+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:47.053944+0000', 'last_peered': '2024-09-11T04:57:47.053944+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:47.053944+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4118, 'log_dups_size': 0, 'ondisk_log_size': 4118, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33725008, 'num_objects': 4117, 'num_object_clones': 0, 'num_object_copies': 16468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1324, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4117, 'num_write_kb': 32935, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 912, 'num_bytes_recovered': 7462912, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2793}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4350", 'reported_seq': 4424, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:48.957814+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:48.957814+0000', 'last_peered': '2024-09-11T04:57:48.957814+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:48.957814+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4350, 'log_dups_size': 0, 'ondisk_log_size': 4350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35625552, 'num_objects': 4349, 'num_object_clones': 0, 'num_object_copies': 17396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4349, 'num_write_kb': 34791, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4280", 'reported_seq': 4329, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:47.015972+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:47.015972+0000', 'last_peered': '2024-09-11T04:57:47.015972+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:47.015972+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4280, 'log_dups_size': 0, 'ondisk_log_size': 4280, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35026080, 'num_objects': 4276, 'num_object_clones': 0, 'num_object_copies': 17104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4278, 'num_write_kb': 34220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4320", 'reported_seq': 4398, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:49.078033+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:49.078033+0000', 'last_peered': '2024-09-11T04:57:49.078033+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:49.078033+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4320, 'log_dups_size': 0, 'ondisk_log_size': 4320, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35361952, 'num_objects': 4317, 'num_object_clones': 0, 'num_object_copies': 17268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4318, 'num_write_kb': 34541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1999}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:49.980 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4288", 'reported_seq': 4351, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:46.275867+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:46.275867+0000', 'last_peered': '2024-09-11T04:57:46.275867+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:46.275867+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4288, 'log_dups_size': 0, 'ondisk_log_size': 4288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35083424, 'num_objects': 4283, 'num_object_clones': 0, 'num_object_copies': 17132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4283, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4286, 'num_write_kb': 34276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1972}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:49.981 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:50.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 127 20 69062 69042 4.24649 2.26562 0.0339487 0.0367821 2024-09-11T04:57:50.732 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:50.733 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35971/272096 objects degraded (13.220%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":197,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68024,"data_bytes":557211840,"bytes_used":1777500160,"bytes_avail":384769556480,"bytes_total":386547056640,"degraded_objects":35971,"degraded_total":272096,"degraded_ratio":0.13219966482417969,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":75743,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"write_bytes_sec":2466112,"read_op_per_sec":0,"write_op_per_sec":301},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:57:50.761 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:50.762 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:51.430 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:51.430 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:51.453 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-11T04:57:49.649599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":557211840,"num_objects":68024,"num_object_clones":0,"num_object_copies":272096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68051,"num_write_kb":544290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68078,"ondisk_log_size":68078,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1735840,"kb_used_data":943328,"kb_used_omap":14,"kb_used_meta":792433,"kb_avail":375751520,"statfs":{"total":386547056640,"available":384769556480,"internally_reserved":0,"allocated":965967872,"data_stored":965284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":811451958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,2,3,13,8,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29605888,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":28912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005085"},"pg_stats":[{"pgid":"1.f","version":"31'4181","reported_seq":4221,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.068879+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:47.068879+0000","last_peered":"2024-09-11T04:57:47.068879+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:47.068879+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34250752,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4181,"num_write_kb":33448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4220","reported_seq":4282,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.201204+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:46.201204+0000","last_peered":"2024-09-11T04:57:46.201204+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:46.201204+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34534560,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":33740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4096","reported_seq":4171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:49.057352+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:49.057352+0000","last_peered":"2024-09-11T04:57:49.057352+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:49.057352+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4096,"log_dups_size":0,"ondisk_log_size":4096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528400,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":32750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4197","reported_seq":4260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.352087+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:46.352087+0000","last_peered":"2024-09-11T04:57:46.352087+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:46.352087+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34346144,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4242","reported_seq":4296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.072851+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:47.072851+0000","last_peered":"2024-09-11T04:57:47.072851+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:47.072851+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34722976,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4240,"num_write_kb":33917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4288","reported_seq":4326,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.952565+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:46.952565+0000","last_peered":"2024-09-11T04:57:46.952565+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:46.952565+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4247","reported_seq":4309,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.297611+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:46.297611+0000","last_peered":"2024-09-11T04:57:46.297611+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:46.297611+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34739360,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4245,"num_write_kb":33940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1924}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4356","reported_seq":4418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.090492+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:46.090492+0000","last_peered":"2024-09-11T04:57:46.090492+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:46.090492+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4356,"log_dups_size":0,"ondisk_log_size":4356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35640480,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4354,"num_write_kb":34820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4319","reported_seq":4375,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.035986+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:47.035986+0000","last_peered":"2024-09-11T04:57:47.035986+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:47.035986+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4319,"log_dups_size":0,"ondisk_log_size":4319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35309888,"num_objects":4311,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4316,"num_write_kb":34505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4286","reported_seq":4363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:48.972235+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:48.972235+0000","last_peered":"2024-09-11T04:57:48.972235+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:48.972235+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35091616,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4284,"num_write_kb":34270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1966}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4290","reported_seq":4345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.015814+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:47.015814+0000","last_peered":"2024-09-11T04:57:47.015814+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:47.015814+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35098352,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":34284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1962}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4118","reported_seq":5977,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.053944+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:47.053944+0000","last_peered":"2024-09-11T04:57:47.053944+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:47.053944+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33725008,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":32935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4350","reported_seq":4424,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:48.957814+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:48.957814+0000","last_peered":"2024-09-11T04:57:48.957814+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:48.957814+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35625552,"num_objects":4349,"num_object_clones":0,"num_object_copies":17396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4349,"num_write_kb":34791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4280","reported_seq":4329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:47.015972+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:47.015972+0000","last_peered":"2024-09-11T04:57:47.015972+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:47.015972+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35026080,"num_objects":4276,"num_object_clones":0,"num_object_copies":17104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":34220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4320","reported_seq":4398,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:49.078033+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:49.078033+0000","last_peered":"2024-09-11T04:57:49.078033+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:49.078033+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35361952,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4318,"num_write_kb":34541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1999}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4288","reported_seq":4351,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:46.275867+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:46.275867+0000","last_peered":"2024-09-11T04:57:46.275867+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:46.275867+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35083424,"num_objects":4283,"num_object_clones":0,"num_object_copies":17132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4286,"num_write_kb":34276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1972}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":557211840,"num_objects":68024,"num_object_clones":0,"num_object_copies":272096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68051,"num_write_kb":544290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":7462912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":967680000,"data_stored":967680000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68078,"ondisk_log_size":68078,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492304,"kb_used_data":271808,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93879536,"statfs":{"total":96636764160,"available":96132644864,"internally_reserved":0,"allocated":278331392,"data_stored":278160582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485508,"kb_used_data":273844,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93886332,"statfs":{"total":96636764160,"available":96139603968,"internally_reserved":0,"allocated":280416256,"data_stored":280245446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,8,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":275224,"kb_used_data":126984,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94096616,"statfs":{"total":96636764160,"available":96354934784,"internally_reserved":0,"allocated":130031616,"data_stored":129860806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":151776978},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482804,"kb_used_data":270692,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93889036,"statfs":{"total":96636764160,"available":96142372864,"internally_reserved":0,"allocated":277188608,"data_stored":277017798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277233664,"data_stored":277233664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131186688,"data_stored":131186688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280989696,"data_stored":280989696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278269952,"data_stored":278269952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:51.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 128 20 69364 69344 4.23175 2.35938 0.0230862 0.0369042 2024-09-11T04:57:52.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 129 20 69694 69674 4.21893 2.57812 0.0718106 0.0370194 2024-09-11T04:57:53.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 130 20 70035 70015 4.20697 2.66406 0.070582 0.0371289 2024-09-11T04:57:54.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:54.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 131 20 70349 70329 4.19358 2.45312 0.0431744 0.0372498 2024-09-11T04:57:55.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:55.088 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-11T04:57:53.651372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":567165120,"num_objects":69239,"num_object_clones":0,"num_object_copies":276956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69266,"num_write_kb":554010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69293,"ondisk_log_size":69293,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1757420,"kb_used_data":963500,"kb_used_omap":14,"kb_used_meta":793841,"kb_avail":375729940,"statfs":{"total":386547056640,"available":384747458560,"internally_reserved":0,"allocated":986624000,"data_stored":985940760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":812893750},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,14,22,18,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32022528,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3909,"num_write_kb":31272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005388"},"pg_stats":[{"pgid":"1.f","version":"31'4275","reported_seq":4317,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.920813+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:51.920813+0000","last_peered":"2024-09-11T04:57:51.920813+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:51.920813+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35020800,"num_objects":4275,"num_object_clones":0,"num_object_copies":17100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4275,"num_write_kb":34200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4343","reported_seq":4405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.315635+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:51.315635+0000","last_peered":"2024-09-11T04:57:51.315635+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:51.315635+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35542176,"num_objects":4339,"num_object_clones":0,"num_object_copies":17356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4341,"num_write_kb":34724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4096","reported_seq":4171,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:49.057352+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:49.057352+0000","last_peered":"2024-09-11T04:57:49.057352+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:49.057352+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4096,"log_dups_size":0,"ondisk_log_size":4096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528400,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":32750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4297","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.274602+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:51.274602+0000","last_peered":"2024-09-11T04:57:51.274602+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:51.274602+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4297,"log_dups_size":0,"ondisk_log_size":4297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35165344,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4337","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.930083+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:51.930083+0000","last_peered":"2024-09-11T04:57:51.930083+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:51.930083+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35501216,"num_objects":4334,"num_object_clones":0,"num_object_copies":17336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4385","reported_seq":4426,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.100554+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:52.100554+0000","last_peered":"2024-09-11T04:57:52.100554+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:52.100554+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35921920,"num_objects":4385,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4346","reported_seq":4408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.379763+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:51.379763+0000","last_peered":"2024-09-11T04:57:51.379763+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:51.379763+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550368,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4344,"num_write_kb":34732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4449","reported_seq":4511,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.164402+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:51.164402+0000","last_peered":"2024-09-11T04:57:51.164402+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:51.164402+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402336,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4447,"num_write_kb":35564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4421","reported_seq":4477,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.093707+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:52.093707+0000","last_peered":"2024-09-11T04:57:52.093707+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:52.093707+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36145472,"num_objects":4413,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4286","reported_seq":4363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:48.972235+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:48.972235+0000","last_peered":"2024-09-11T04:57:48.972235+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:48.972235+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35091616,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4284,"num_write_kb":34270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1966}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4396","reported_seq":4452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.008467+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:52.008467+0000","last_peered":"2024-09-11T04:57:52.008467+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:52.008467+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35966704,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":35132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4230","reported_seq":6185,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.134494+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:52.134494+0000","last_peered":"2024-09-11T04:57:52.134494+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:52.134494+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34642512,"num_objects":4229,"num_object_clones":0,"num_object_copies":16916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4229,"num_write_kb":33831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2954}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4350","reported_seq":4424,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:48.957814+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:48.957814+0000","last_peered":"2024-09-11T04:57:48.957814+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:48.957814+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35625552,"num_objects":4349,"num_object_clones":0,"num_object_copies":17396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4349,"num_write_kb":34791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4370","reported_seq":4419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.092460+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:52.092460+0000","last_peered":"2024-09-11T04:57:52.092460+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:52.092460+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35763360,"num_objects":4366,"num_object_clones":0,"num_object_copies":17464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4368,"num_write_kb":34940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4320","reported_seq":4398,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:49.078033+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:49.078033+0000","last_peered":"2024-09-11T04:57:49.078033+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:49.078033+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35361952,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4318,"num_write_kb":34541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1999}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4392","reported_seq":4455,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.266211+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:51.266211+0000","last_peered":"2024-09-11T04:57:51.266211+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:51.266211+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935392,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4390,"num_write_kb":35108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":567165120,"num_objects":69239,"num_object_clones":0,"num_object_copies":276956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69266,"num_write_kb":554010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":987193344,"data_stored":987193344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69293,"ondisk_log_size":69293,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485952,"kb_used_data":277936,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93885888,"statfs":{"total":96636764160,"available":96139149312,"internally_reserved":0,"allocated":284606464,"data_stored":284435654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":212987600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,6,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485508,"kb_used_data":273844,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93886332,"statfs":{"total":96636764160,"available":96139603968,"internally_reserved":0,"allocated":280416256,"data_stored":280245446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,8,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292724,"kb_used_data":134756,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94079116,"statfs":{"total":96636764160,"available":96337014784,"internally_reserved":0,"allocated":137990144,"data_stored":137819334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161738450},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493236,"kb_used_data":276964,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93878604,"statfs":{"total":96636764160,"available":96131690496,"internally_reserved":0,"allocated":283611136,"data_stored":283440326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8,13,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283717632,"data_stored":283717632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137797632,"data_stored":137797632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280989696,"data_stored":280989696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284688384,"data_stored":284688384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:55.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:55.673 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:55.674 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:55.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 132 20 70634 70614 4.17868 2.22656 0.0650298 0.0373812 2024-09-11T04:57:55.705 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-11T04:57:55.652200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570360000,"num_objects":69629,"num_object_clones":0,"num_object_copies":278516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69656,"num_write_kb":557130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69683,"ondisk_log_size":69683,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1767700,"kb_used_data":969620,"kb_used_omap":14,"kb_used_meta":798001,"kb_avail":375719660,"statfs":{"total":386547056640,"available":384736931840,"internally_reserved":0,"allocated":992890880,"data_stored":992207640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":817153590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,15,14,18,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29450240,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3595,"num_write_kb":28760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005269"},"pg_stats":[{"pgid":"1.f","version":"31'4275","reported_seq":4317,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.920813+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:51.920813+0000","last_peered":"2024-09-11T04:57:51.920813+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:51.920813+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35020800,"num_objects":4275,"num_object_clones":0,"num_object_copies":17100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4275,"num_write_kb":34200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4343","reported_seq":4405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.315635+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:51.315635+0000","last_peered":"2024-09-11T04:57:51.315635+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:51.315635+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35542176,"num_objects":4339,"num_object_clones":0,"num_object_copies":17356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4341,"num_write_kb":34724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4191","reported_seq":4266,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:53.929046+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:53.929046+0000","last_peered":"2024-09-11T04:57:53.929046+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:53.929046+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34306640,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4190,"num_write_kb":33510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4297","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.274602+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:51.274602+0000","last_peered":"2024-09-11T04:57:51.274602+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:51.274602+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4297,"log_dups_size":0,"ondisk_log_size":4297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35165344,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4337","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.930083+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:51.930083+0000","last_peered":"2024-09-11T04:57:51.930083+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:51.930083+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35501216,"num_objects":4334,"num_object_clones":0,"num_object_copies":17336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4385","reported_seq":4426,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.100554+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:52.100554+0000","last_peered":"2024-09-11T04:57:52.100554+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:52.100554+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35921920,"num_objects":4385,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4346","reported_seq":4408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.379763+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:51.379763+0000","last_peered":"2024-09-11T04:57:51.379763+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:51.379763+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550368,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4344,"num_write_kb":34732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4449","reported_seq":4511,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.164402+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:51.164402+0000","last_peered":"2024-09-11T04:57:51.164402+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:51.164402+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402336,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4447,"num_write_kb":35564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4421","reported_seq":4477,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.093707+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:52.093707+0000","last_peered":"2024-09-11T04:57:52.093707+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:52.093707+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36145472,"num_objects":4413,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4371","reported_seq":4448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:54.127892+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:54.127892+0000","last_peered":"2024-09-11T04:57:54.127892+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:54.127892+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35787936,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4369,"num_write_kb":34950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4396","reported_seq":4452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.008467+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:52.008467+0000","last_peered":"2024-09-11T04:57:52.008467+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:52.008467+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35966704,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":35132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4230","reported_seq":6185,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.134494+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:52.134494+0000","last_peered":"2024-09-11T04:57:52.134494+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:52.134494+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34642512,"num_objects":4229,"num_object_clones":0,"num_object_copies":16916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4229,"num_write_kb":33831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2954}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4470","reported_seq":4544,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:54.077165+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:54.077165+0000","last_peered":"2024-09-11T04:57:54.077165+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:54.077165+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36608592,"num_objects":4469,"num_object_clones":0,"num_object_copies":17876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":35751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2137}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4370","reported_seq":4419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.092460+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:52.092460+0000","last_peered":"2024-09-11T04:57:52.092460+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:52.092460+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35763360,"num_objects":4366,"num_object_clones":0,"num_object_copies":17464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4368,"num_write_kb":34940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4410","reported_seq":4488,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:54.069258+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:54.069258+0000","last_peered":"2024-09-11T04:57:54.069258+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:54.069258+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36099232,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":35261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2089}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4392","reported_seq":4455,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.266211+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:51.266211+0000","last_peered":"2024-09-11T04:57:51.266211+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:51.266211+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935392,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4390,"num_write_kb":35108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":570360000,"num_objects":69629,"num_object_clones":0,"num_object_copies":278516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69656,"num_write_kb":557130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":993644544,"data_stored":993644544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69683,"ondisk_log_size":69683,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485952,"kb_used_data":277936,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93885888,"statfs":{"total":96636764160,"available":96139149312,"internally_reserved":0,"allocated":284606464,"data_stored":284435654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":212987600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,6,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495788,"kb_used_data":279964,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93876052,"statfs":{"total":96636764160,"available":96129077248,"internally_reserved":0,"allocated":286683136,"data_stored":286512326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292724,"kb_used_data":134756,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94079116,"statfs":{"total":96636764160,"available":96337014784,"internally_reserved":0,"allocated":137990144,"data_stored":137819334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161738450},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493236,"kb_used_data":276964,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93878604,"statfs":{"total":96636764160,"available":96131690496,"internally_reserved":0,"allocated":283611136,"data_stored":283440326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8,13,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283717632,"data_stored":283717632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137797632,"data_stored":137797632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287440896,"data_stored":287440896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284688384,"data_stored":284688384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:55.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:57:55.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4275", 'reported_seq': 4317, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:51.920813+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:51.920813+0000', 'last_peered': '2024-09-11T04:57:51.920813+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:51.920813+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4275, 'log_dups_size': 0, 'ondisk_log_size': 4275, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35020800, 'num_objects': 4275, 'num_object_clones': 0, 'num_object_copies': 17100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4275, 'num_write_kb': 34200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1963}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:55.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4343", 'reported_seq': 4405, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:51.315635+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:51.315635+0000', 'last_peered': '2024-09-11T04:57:51.315635+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:51.315635+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4343, 'log_dups_size': 0, 'ondisk_log_size': 4343, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35542176, 'num_objects': 4339, 'num_object_clones': 0, 'num_object_copies': 17356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4341, 'num_write_kb': 34724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2060}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4191", 'reported_seq': 4266, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:53.929046+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:53.929046+0000', 'last_peered': '2024-09-11T04:57:53.929046+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:53.929046+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4191, 'log_dups_size': 0, 'ondisk_log_size': 4191, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34306640, 'num_objects': 4188, 'num_object_clones': 0, 'num_object_copies': 16752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4190, 'num_write_kb': 33510, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1968}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4297", 'reported_seq': 4360, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:51.274602+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:51.274602+0000', 'last_peered': '2024-09-11T04:57:51.274602+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:51.274602+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4297, 'log_dups_size': 0, 'ondisk_log_size': 4297, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35165344, 'num_objects': 4293, 'num_object_clones': 0, 'num_object_copies': 17172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4295, 'num_write_kb': 34349, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2015}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4337", 'reported_seq': 4391, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:51.930083+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:51.930083+0000', 'last_peered': '2024-09-11T04:57:51.930083+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:51.930083+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4337, 'log_dups_size': 0, 'ondisk_log_size': 4337, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35501216, 'num_objects': 4334, 'num_object_clones': 0, 'num_object_copies': 17336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4335, 'num_write_kb': 34677, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2084}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4385", 'reported_seq': 4426, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:52.100554+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:52.100554+0000', 'last_peered': '2024-09-11T04:57:52.100554+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:52.100554+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4385, 'log_dups_size': 0, 'ondisk_log_size': 4385, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35921920, 'num_objects': 4385, 'num_object_clones': 0, 'num_object_copies': 17540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4385, 'num_write_kb': 35080, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2064}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:55.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4346", 'reported_seq': 4408, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:51.379763+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:51.379763+0000', 'last_peered': '2024-09-11T04:57:51.379763+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:51.379763+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4346, 'log_dups_size': 0, 'ondisk_log_size': 4346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35550368, 'num_objects': 4340, 'num_object_clones': 0, 'num_object_copies': 17360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4344, 'num_write_kb': 34732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2023}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4449", 'reported_seq': 4511, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:51.164402+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:51.164402+0000', 'last_peered': '2024-09-11T04:57:51.164402+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:51.164402+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4449, 'log_dups_size': 0, 'ondisk_log_size': 4449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36402336, 'num_objects': 4444, 'num_object_clones': 0, 'num_object_copies': 17776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4447, 'num_write_kb': 35564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4421", 'reported_seq': 4477, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:52.093707+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:52.093707+0000', 'last_peered': '2024-09-11T04:57:52.093707+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:52.093707+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4421, 'log_dups_size': 0, 'ondisk_log_size': 4421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36145472, 'num_objects': 4413, 'num_object_clones': 0, 'num_object_copies': 17652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4418, 'num_write_kb': 35321, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2019}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4371", 'reported_seq': 4448, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:54.127892+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:54.127892+0000', 'last_peered': '2024-09-11T04:57:54.127892+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:54.127892+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4371, 'log_dups_size': 0, 'ondisk_log_size': 4371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35787936, 'num_objects': 4369, 'num_object_clones': 0, 'num_object_copies': 17476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4369, 'num_write_kb': 34950, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2051}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4396", 'reported_seq': 4452, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:52.008467+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:52.008467+0000', 'last_peered': '2024-09-11T04:57:52.008467+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:52.008467+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4396, 'log_dups_size': 0, 'ondisk_log_size': 4396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35966704, 'num_objects': 4391, 'num_object_clones': 0, 'num_object_copies': 17564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4393, 'num_write_kb': 35132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2068}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:55.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4230", 'reported_seq': 6185, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:52.134494+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:52.134494+0000', 'last_peered': '2024-09-11T04:57:52.134494+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:52.134494+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4230, 'log_dups_size': 0, 'ondisk_log_size': 4230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34642512, 'num_objects': 4229, 'num_object_clones': 0, 'num_object_copies': 16916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4229, 'num_write_kb': 33831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 961, 'num_bytes_recovered': 7864320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4470", 'reported_seq': 4544, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:54.077165+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:54.077165+0000', 'last_peered': '2024-09-11T04:57:54.077165+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:54.077165+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4470, 'log_dups_size': 0, 'ondisk_log_size': 4470, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36608592, 'num_objects': 4469, 'num_object_clones': 0, 'num_object_copies': 17876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4469, 'num_write_kb': 35751, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2137}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4370", 'reported_seq': 4419, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:52.092460+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:52.092460+0000', 'last_peered': '2024-09-11T04:57:52.092460+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:52.092460+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4370, 'log_dups_size': 0, 'ondisk_log_size': 4370, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35763360, 'num_objects': 4366, 'num_object_clones': 0, 'num_object_copies': 17464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4368, 'num_write_kb': 34940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4410", 'reported_seq': 4488, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:54.069258+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:54.069258+0000', 'last_peered': '2024-09-11T04:57:54.069258+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:54.069258+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4410, 'log_dups_size': 0, 'ondisk_log_size': 4410, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36099232, 'num_objects': 4407, 'num_object_clones': 0, 'num_object_copies': 17628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4408, 'num_write_kb': 35261, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:57:55.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4392", 'reported_seq': 4455, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:51.266211+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:51.266211+0000', 'last_peered': '2024-09-11T04:57:51.266211+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:51.266211+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4392, 'log_dups_size': 0, 'ondisk_log_size': 4392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35935392, 'num_objects': 4387, 'num_object_clones': 0, 'num_object_copies': 17548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4390, 'num_write_kb': 35108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2076}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:57:55.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:57:56.505 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:56.505 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35922/278516 objects degraded (12.898%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69629,"data_bytes":570360000,"bytes_used":1810124800,"bytes_avail":384736931840,"bytes_total":386547056640,"degraded_objects":35922,"degraded_total":278516,"degraded_ratio":0.12897643223369573,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":71648,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"write_bytes_sec":2453109,"read_op_per_sec":0,"write_op_per_sec":299},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:57:56.527 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:57:56.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:57:56.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 133 20 70919 70899 4.164 2.22656 0.0871832 0.0375119 2024-09-11T04:57:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:57:57.197 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:57:57.229 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-11T04:57:55.652200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570360000,"num_objects":69629,"num_object_clones":0,"num_object_copies":278516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69656,"num_write_kb":557130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69683,"ondisk_log_size":69683,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1767700,"kb_used_data":969620,"kb_used_omap":14,"kb_used_meta":798001,"kb_avail":375719660,"statfs":{"total":386547056640,"available":384736931840,"internally_reserved":0,"allocated":992890880,"data_stored":992207640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":817153590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,15,14,18,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29450240,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3595,"num_write_kb":28760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005269"},"pg_stats":[{"pgid":"1.f","version":"31'4275","reported_seq":4317,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.920813+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:51.920813+0000","last_peered":"2024-09-11T04:57:51.920813+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:51.920813+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35020800,"num_objects":4275,"num_object_clones":0,"num_object_copies":17100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4275,"num_write_kb":34200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4343","reported_seq":4405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.315635+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:51.315635+0000","last_peered":"2024-09-11T04:57:51.315635+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:51.315635+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35542176,"num_objects":4339,"num_object_clones":0,"num_object_copies":17356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4341,"num_write_kb":34724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4191","reported_seq":4266,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:53.929046+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:53.929046+0000","last_peered":"2024-09-11T04:57:53.929046+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:53.929046+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34306640,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4190,"num_write_kb":33510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4297","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.274602+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:51.274602+0000","last_peered":"2024-09-11T04:57:51.274602+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:51.274602+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4297,"log_dups_size":0,"ondisk_log_size":4297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35165344,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4337","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.930083+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:51.930083+0000","last_peered":"2024-09-11T04:57:51.930083+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:51.930083+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35501216,"num_objects":4334,"num_object_clones":0,"num_object_copies":17336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":34677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4385","reported_seq":4426,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.100554+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:52.100554+0000","last_peered":"2024-09-11T04:57:52.100554+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:52.100554+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35921920,"num_objects":4385,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4346","reported_seq":4408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.379763+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:51.379763+0000","last_peered":"2024-09-11T04:57:51.379763+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:51.379763+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35550368,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4344,"num_write_kb":34732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4449","reported_seq":4511,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.164402+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:51.164402+0000","last_peered":"2024-09-11T04:57:51.164402+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:51.164402+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402336,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4447,"num_write_kb":35564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4421","reported_seq":4477,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.093707+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:52.093707+0000","last_peered":"2024-09-11T04:57:52.093707+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:52.093707+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36145472,"num_objects":4413,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4371","reported_seq":4448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:54.127892+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:54.127892+0000","last_peered":"2024-09-11T04:57:54.127892+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:54.127892+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35787936,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4369,"num_write_kb":34950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4396","reported_seq":4452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.008467+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:52.008467+0000","last_peered":"2024-09-11T04:57:52.008467+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:52.008467+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35966704,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":35132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4230","reported_seq":6185,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.134494+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:52.134494+0000","last_peered":"2024-09-11T04:57:52.134494+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:52.134494+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34642512,"num_objects":4229,"num_object_clones":0,"num_object_copies":16916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4229,"num_write_kb":33831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2954}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4470","reported_seq":4544,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:54.077165+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:54.077165+0000","last_peered":"2024-09-11T04:57:54.077165+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:54.077165+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36608592,"num_objects":4469,"num_object_clones":0,"num_object_copies":17876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":35751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2137}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4370","reported_seq":4419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:52.092460+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:52.092460+0000","last_peered":"2024-09-11T04:57:52.092460+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:52.092460+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35763360,"num_objects":4366,"num_object_clones":0,"num_object_copies":17464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4368,"num_write_kb":34940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4410","reported_seq":4488,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:54.069258+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:54.069258+0000","last_peered":"2024-09-11T04:57:54.069258+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:54.069258+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36099232,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":35261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2089}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4392","reported_seq":4455,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:51.266211+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:51.266211+0000","last_peered":"2024-09-11T04:57:51.266211+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:51.266211+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935392,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4390,"num_write_kb":35108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":570360000,"num_objects":69629,"num_object_clones":0,"num_object_copies":278516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69656,"num_write_kb":557130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":993644544,"data_stored":993644544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69683,"ondisk_log_size":69683,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485952,"kb_used_data":277936,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93885888,"statfs":{"total":96636764160,"available":96139149312,"internally_reserved":0,"allocated":284606464,"data_stored":284435654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":212987600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,6,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495788,"kb_used_data":279964,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93876052,"statfs":{"total":96636764160,"available":96129077248,"internally_reserved":0,"allocated":286683136,"data_stored":286512326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292724,"kb_used_data":134756,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94079116,"statfs":{"total":96636764160,"available":96337014784,"internally_reserved":0,"allocated":137990144,"data_stored":137819334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161738450},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493236,"kb_used_data":276964,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93878604,"statfs":{"total":96636764160,"available":96131690496,"internally_reserved":0,"allocated":283611136,"data_stored":283440326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8,13,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283717632,"data_stored":283717632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137797632,"data_stored":137797632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287440896,"data_stored":287440896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284688384,"data_stored":284688384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:57:57.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 134 20 71226 71206 4.15082 2.39844 0.0509054 0.0376298 2024-09-11T04:57:58.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 135 20 71549 71529 4.13877 2.52344 0.0916301 0.0377378 2024-09-11T04:57:59.705 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 136 20 71860 71840 4.12613 2.42969 0.0536847 0.0378552 2024-09-11T04:58:00.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:00.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 137 20 72174 72154 4.11392 2.45312 0.140463 0.0379655 2024-09-11T04:58:00.819 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:00.819 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:00.852 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-11T04:57:59.653851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583049408,"num_objects":71178,"num_object_clones":0,"num_object_copies":284712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71205,"num_write_kb":569522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71232,"ondisk_log_size":71232,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1849280,"kb_used_data":994304,"kb_used_omap":14,"kb_used_meta":854897,"kb_avail":375638080,"statfs":{"total":386547056640,"available":384653393920,"internally_reserved":0,"allocated":1018167296,"data_stored":1017484056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":875415094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,6,11,8,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005064"},"pg_stats":[{"pgid":"1.f","version":"31'4373","reported_seq":4415,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.088695+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:57.088695+0000","last_peered":"2024-09-11T04:57:57.088695+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:57.088695+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35823616,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4373,"num_write_kb":34984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4455","reported_seq":4516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.265934+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:56.265934+0000","last_peered":"2024-09-11T04:57:56.265934+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:56.265934+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36459680,"num_objects":4451,"num_object_clones":0,"num_object_copies":17804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":35620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4298","reported_seq":4373,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.180233+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:59.180233+0000","last_peered":"2024-09-11T04:57:59.180233+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:59.180233+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35183184,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4297,"num_write_kb":34366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4391","reported_seq":4453,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.355408+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:56.355408+0000","last_peered":"2024-09-11T04:57:56.355408+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:56.355408+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935392,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4428","reported_seq":4482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.984858+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:56.984858+0000","last_peered":"2024-09-11T04:57:56.984858+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:56.984858+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36246688,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":35405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4491","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.186275+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:57.186275+0000","last_peered":"2024-09-11T04:57:57.186275+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:57.186275+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36790272,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4426","reported_seq":4489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.380521+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:56.380521+0000","last_peered":"2024-09-11T04:57:56.380521+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:56.380521+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36205728,"num_objects":4420,"num_object_clones":0,"num_object_copies":17680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4424,"num_write_kb":35372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2103}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4540","reported_seq":4602,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.378981+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:56.378981+0000","last_peered":"2024-09-11T04:57:56.378981+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:56.378981+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37147808,"num_objects":4535,"num_object_clones":0,"num_object_copies":18140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4538,"num_write_kb":36292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4499","reported_seq":4555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.155461+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:57.155461+0000","last_peered":"2024-09-11T04:57:57.155461+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:57.155461+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36784448,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2097}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4480","reported_seq":4557,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.064719+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:59.064719+0000","last_peered":"2024-09-11T04:57:59.064719+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:59.064719+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680864,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4478,"num_write_kb":35822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4494","reported_seq":4550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.056529+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:57.056529+0000","last_peered":"2024-09-11T04:57:57.056529+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:57.056529+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36769520,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4333","reported_seq":6400,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.189108+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:57.189108+0000","last_peered":"2024-09-11T04:57:57.189108+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:57.189108+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4333,"log_dups_size":0,"ondisk_log_size":4333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486288,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":34655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3112}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4565","reported_seq":4632,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.169215+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:59.169215+0000","last_peered":"2024-09-11T04:57:59.169215+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:59.169215+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37386832,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":36511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4459","reported_seq":4508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.195599+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:57.195599+0000","last_peered":"2024-09-11T04:57:57.195599+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:57.195599+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4459,"log_dups_size":0,"ondisk_log_size":4459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36492448,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4457,"num_write_kb":35652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4502","reported_seq":4580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.095745+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:59.095745+0000","last_peered":"2024-09-11T04:57:59.095745+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:59.095745+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36852896,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4500,"num_write_kb":35997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4498","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.391995+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:56.391995+0000","last_peered":"2024-09-11T04:57:56.391995+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:56.391995+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36803744,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583049408,"num_objects":71178,"num_object_clones":0,"num_object_copies":284712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71205,"num_write_kb":569522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1019289600,"data_stored":1019289600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71232,"ondisk_log_size":71232,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514216,"kb_used_data":283992,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93857624,"statfs":{"total":96636764160,"available":96110206976,"internally_reserved":0,"allocated":290807808,"data_stored":290636998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":235728592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528148,"kb_used_data":285956,"kb_used_omap":1,"kb_used_meta":242174,"kb_avail":93843692,"statfs":{"total":96636764160,"available":96095940608,"internally_reserved":0,"allocated":292818944,"data_stored":292648134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":247986629},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303244,"kb_used_data":141116,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94068596,"statfs":{"total":96636764160,"available":96326242304,"internally_reserved":0,"allocated":144502784,"data_stored":144331974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":165998290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":503672,"kb_used_data":283240,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93868168,"statfs":{"total":96636764160,"available":96121004032,"internally_reserved":0,"allocated":290037760,"data_stored":289866950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225701583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290131968,"data_stored":290131968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144400384,"data_stored":144400384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293675008,"data_stored":293675008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291082240,"data_stored":291082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:00.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:01.492 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:01.492 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:01.521 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-11T04:57:59.653851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583049408,"num_objects":71178,"num_object_clones":0,"num_object_copies":284712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71205,"num_write_kb":569522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71232,"ondisk_log_size":71232,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1849280,"kb_used_data":994304,"kb_used_omap":14,"kb_used_meta":854897,"kb_avail":375638080,"statfs":{"total":386547056640,"available":384653393920,"internally_reserved":0,"allocated":1018167296,"data_stored":1017484056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":875415094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,6,11,8,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28925952,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":28248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005064"},"pg_stats":[{"pgid":"1.f","version":"31'4373","reported_seq":4415,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.088695+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:57.088695+0000","last_peered":"2024-09-11T04:57:57.088695+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:57.088695+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35823616,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4373,"num_write_kb":34984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4455","reported_seq":4516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.265934+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:57:56.265934+0000","last_peered":"2024-09-11T04:57:56.265934+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:57:56.265934+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36459680,"num_objects":4451,"num_object_clones":0,"num_object_copies":17804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":35620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4298","reported_seq":4373,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.180233+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:59.180233+0000","last_peered":"2024-09-11T04:57:59.180233+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:59.180233+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35183184,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4297,"num_write_kb":34366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4391","reported_seq":4453,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.355408+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:57:56.355408+0000","last_peered":"2024-09-11T04:57:56.355408+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:57:56.355408+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935392,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4428","reported_seq":4482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.984858+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:56.984858+0000","last_peered":"2024-09-11T04:57:56.984858+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:56.984858+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36246688,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":35405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4491","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.186275+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:57.186275+0000","last_peered":"2024-09-11T04:57:57.186275+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:57.186275+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36790272,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4426","reported_seq":4489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.380521+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:57:56.380521+0000","last_peered":"2024-09-11T04:57:56.380521+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:57:56.380521+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36205728,"num_objects":4420,"num_object_clones":0,"num_object_copies":17680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4424,"num_write_kb":35372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2103}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4540","reported_seq":4602,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.378981+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:57:56.378981+0000","last_peered":"2024-09-11T04:57:56.378981+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:57:56.378981+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37147808,"num_objects":4535,"num_object_clones":0,"num_object_copies":18140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4538,"num_write_kb":36292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4499","reported_seq":4555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.155461+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:57.155461+0000","last_peered":"2024-09-11T04:57:57.155461+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:57.155461+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36784448,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2097}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4480","reported_seq":4557,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.064719+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:59.064719+0000","last_peered":"2024-09-11T04:57:59.064719+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:59.064719+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680864,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4478,"num_write_kb":35822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4494","reported_seq":4550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.056529+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:57.056529+0000","last_peered":"2024-09-11T04:57:57.056529+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:57.056529+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36769520,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4333","reported_seq":6400,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.189108+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:57.189108+0000","last_peered":"2024-09-11T04:57:57.189108+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:57.189108+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4333,"log_dups_size":0,"ondisk_log_size":4333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486288,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":34655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3112}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4565","reported_seq":4632,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.169215+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:59.169215+0000","last_peered":"2024-09-11T04:57:59.169215+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:59.169215+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37386832,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":36511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4459","reported_seq":4508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.195599+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:57.195599+0000","last_peered":"2024-09-11T04:57:57.195599+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:57.195599+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4459,"log_dups_size":0,"ondisk_log_size":4459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36492448,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4457,"num_write_kb":35652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4502","reported_seq":4580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.095745+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:59.095745+0000","last_peered":"2024-09-11T04:57:59.095745+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:59.095745+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36852896,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4500,"num_write_kb":35997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4498","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.391995+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:57:56.391995+0000","last_peered":"2024-09-11T04:57:56.391995+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:57:56.391995+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36803744,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583049408,"num_objects":71178,"num_object_clones":0,"num_object_copies":284712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71205,"num_write_kb":569522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1019289600,"data_stored":1019289600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71232,"ondisk_log_size":71232,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514216,"kb_used_data":283992,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93857624,"statfs":{"total":96636764160,"available":96110206976,"internally_reserved":0,"allocated":290807808,"data_stored":290636998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":235728592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528148,"kb_used_data":285956,"kb_used_omap":1,"kb_used_meta":242174,"kb_avail":93843692,"statfs":{"total":96636764160,"available":96095940608,"internally_reserved":0,"allocated":292818944,"data_stored":292648134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":247986629},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303244,"kb_used_data":141116,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94068596,"statfs":{"total":96636764160,"available":96326242304,"internally_reserved":0,"allocated":144502784,"data_stored":144331974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":165998290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":503672,"kb_used_data":283240,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93868168,"statfs":{"total":96636764160,"available":96121004032,"internally_reserved":0,"allocated":290037760,"data_stored":289866950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225701583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290131968,"data_stored":290131968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144400384,"data_stored":144400384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293675008,"data_stored":293675008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291082240,"data_stored":291082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:01.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:01.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4373", 'reported_seq': 4415, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:57.088695+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:57:57.088695+0000', 'last_peered': '2024-09-11T04:57:57.088695+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:57:57.088695+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4373, 'log_dups_size': 0, 'ondisk_log_size': 4373, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35823616, 'num_objects': 4373, 'num_object_clones': 0, 'num_object_copies': 17492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4373, 'num_write_kb': 34984, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2061}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:01.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:01.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4455", 'reported_seq': 4516, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:56.265934+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:57:56.265934+0000', 'last_peered': '2024-09-11T04:57:56.265934+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:57:56.265934+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4455, 'log_dups_size': 0, 'ondisk_log_size': 4455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36459680, 'num_objects': 4451, 'num_object_clones': 0, 'num_object_copies': 17804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 35620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2172}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:01.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:01.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4298", 'reported_seq': 4373, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:59.180233+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:57:59.180233+0000', 'last_peered': '2024-09-11T04:57:59.180233+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:57:59.180233+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4298, 'log_dups_size': 0, 'ondisk_log_size': 4298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35183184, 'num_objects': 4295, 'num_object_clones': 0, 'num_object_copies': 17180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4297, 'num_write_kb': 34366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2075}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:01.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4391", 'reported_seq': 4453, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:56.355408+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:57:56.355408+0000', 'last_peered': '2024-09-11T04:57:56.355408+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:57:56.355408+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4391, 'log_dups_size': 0, 'ondisk_log_size': 4391, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35935392, 'num_objects': 4387, 'num_object_clones': 0, 'num_object_copies': 17548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4389, 'num_write_kb': 35101, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4428", 'reported_seq': 4482, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:56.984858+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:57:56.984858+0000', 'last_peered': '2024-09-11T04:57:56.984858+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:57:56.984858+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4428, 'log_dups_size': 0, 'ondisk_log_size': 4428, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36246688, 'num_objects': 4425, 'num_object_clones': 0, 'num_object_copies': 17700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4426, 'num_write_kb': 35405, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2175}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4491", 'reported_seq': 4531, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:57.186275+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:57:57.186275+0000', 'last_peered': '2024-09-11T04:57:57.186275+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:57:57.186275+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4491, 'log_dups_size': 0, 'ondisk_log_size': 4491, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36790272, 'num_objects': 4491, 'num_object_clones': 0, 'num_object_copies': 17964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4491, 'num_write_kb': 35928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2170}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4426", 'reported_seq': 4489, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:56.380521+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:57:56.380521+0000', 'last_peered': '2024-09-11T04:57:56.380521+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:57:56.380521+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4426, 'log_dups_size': 0, 'ondisk_log_size': 4426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36205728, 'num_objects': 4420, 'num_object_clones': 0, 'num_object_copies': 17680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4424, 'num_write_kb': 35372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2103}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4540", 'reported_seq': 4602, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:56.378981+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:57:56.378981+0000', 'last_peered': '2024-09-11T04:57:56.378981+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:57:56.378981+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4540, 'log_dups_size': 0, 'ondisk_log_size': 4540, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37147808, 'num_objects': 4535, 'num_object_clones': 0, 'num_object_copies': 18140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4535, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4538, 'num_write_kb': 36292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2174}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:01.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4499", 'reported_seq': 4555, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:57.155461+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:57:57.155461+0000', 'last_peered': '2024-09-11T04:57:57.155461+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:57:57.155461+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4499, 'log_dups_size': 0, 'ondisk_log_size': 4499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36784448, 'num_objects': 4491, 'num_object_clones': 0, 'num_object_copies': 17964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4496, 'num_write_kb': 35945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2097}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4480", 'reported_seq': 4557, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:59.064719+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:57:59.064719+0000', 'last_peered': '2024-09-11T04:57:59.064719+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:57:59.064719+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4480, 'log_dups_size': 0, 'ondisk_log_size': 4480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36680864, 'num_objects': 4478, 'num_object_clones': 0, 'num_object_copies': 17912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4478, 'num_write_kb': 35822, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2160}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4494", 'reported_seq': 4550, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:57.056529+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:57:57.056529+0000', 'last_peered': '2024-09-11T04:57:57.056529+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:57:57.056529+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4494, 'log_dups_size': 0, 'ondisk_log_size': 4494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36769520, 'num_objects': 4489, 'num_object_clones': 0, 'num_object_copies': 17956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4491, 'num_write_kb': 35916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2166}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4333", 'reported_seq': 6400, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:57.189108+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:57:57.189108+0000', 'last_peered': '2024-09-11T04:57:57.189108+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:57:57.189108+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4333, 'log_dups_size': 0, 'ondisk_log_size': 4333, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35486288, 'num_objects': 4332, 'num_object_clones': 0, 'num_object_copies': 17328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4332, 'num_write_kb': 34655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1016, 'num_bytes_recovered': 8314880, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3112}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4565", 'reported_seq': 4632, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:59.169215+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:57:59.169215+0000', 'last_peered': '2024-09-11T04:57:59.169215+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:57:59.169215+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4565, 'log_dups_size': 0, 'ondisk_log_size': 4565, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37386832, 'num_objects': 4564, 'num_object_clones': 0, 'num_object_copies': 18256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4564, 'num_write_kb': 36511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:01.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4459", 'reported_seq': 4508, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:57.195599+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:57:57.195599+0000', 'last_peered': '2024-09-11T04:57:57.195599+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:57:57.195599+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4459, 'log_dups_size': 0, 'ondisk_log_size': 4459, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36492448, 'num_objects': 4455, 'num_object_clones': 0, 'num_object_copies': 17820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4457, 'num_write_kb': 35652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:01.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:01.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4502", 'reported_seq': 4580, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:59.095745+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:57:59.095745+0000', 'last_peered': '2024-09-11T04:57:59.095745+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:57:59.095745+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4502, 'log_dups_size': 0, 'ondisk_log_size': 4502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36852896, 'num_objects': 4499, 'num_object_clones': 0, 'num_object_copies': 17996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4500, 'num_write_kb': 35997, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2181}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:01.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:01.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4498", 'reported_seq': 4561, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:57:56.391995+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:57:56.391995+0000', 'last_peered': '2024-09-11T04:57:56.391995+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:57:56.391995+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4498, 'log_dups_size': 0, 'ondisk_log_size': 4498, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36803744, 'num_objects': 4493, 'num_object_clones': 0, 'num_object_copies': 17972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4496, 'num_write_kb': 35956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2182}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:01.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:01.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 138 20 72462 72442 4.10041 2.25 0.0777791 0.0380896 2024-09-11T04:58:02.317 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:02.318 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35867/286548 objects degraded (12.517%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":209,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71637,"data_bytes":586809536,"bytes_used":1905778688,"bytes_avail":384641277952,"bytes_total":386547056640,"degraded_objects":35867,"degraded_total":286548,"degraded_ratio":0.12516925611066906,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":70967,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":851968,"num_keys_recovered":0,"write_bytes_sec":2465451,"read_op_per_sec":0,"write_op_per_sec":300},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:02.348 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:02.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:02.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 139 20 72754 72734 4.08732 2.28125 0.032808 0.0382157 2024-09-11T04:58:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:03.053 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-11T04:58:01.654578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586809536,"num_objects":71637,"num_object_clones":0,"num_object_copies":286548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71664,"num_write_kb":573194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71691,"ondisk_log_size":71691,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1861112,"kb_used_data":1000568,"kb_used_omap":14,"kb_used_meta":860465,"kb_avail":375626248,"statfs":{"total":386547056640,"available":384641277952,"internally_reserved":0,"allocated":1024581632,"data_stored":1023898392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":881116726},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,6,7,3,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29597696,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004978"},"pg_stats":[{"pgid":"1.f","version":"31'4373","reported_seq":4415,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.088695+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:57:57.088695+0000","last_peered":"2024-09-11T04:57:57.088695+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:57:57.088695+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35823616,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4373,"num_write_kb":34984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4547","reported_seq":4609,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.409456+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:01.409456+0000","last_peered":"2024-09-11T04:58:01.409456+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:01.409456+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37213344,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4545,"num_write_kb":36356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4298","reported_seq":4373,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.180233+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:57:59.180233+0000","last_peered":"2024-09-11T04:57:59.180233+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:57:59.180233+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35183184,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4297,"num_write_kb":34366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4484","reported_seq":4547,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.376148+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:01.376148+0000","last_peered":"2024-09-11T04:58:01.376148+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:01.376148+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697248,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":35845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2202}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4428","reported_seq":4482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:56.984858+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:57:56.984858+0000","last_peered":"2024-09-11T04:57:56.984858+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:57:56.984858+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36246688,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":35405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4491","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.186275+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:57:57.186275+0000","last_peered":"2024-09-11T04:57:57.186275+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:57:57.186275+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36790272,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4515","reported_seq":4578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.408659+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:01.408659+0000","last_peered":"2024-09-11T04:58:01.408659+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:01.408659+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934816,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4513,"num_write_kb":36084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4632","reported_seq":4694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.385731+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:01.385731+0000","last_peered":"2024-09-11T04:58:01.385731+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:01.385731+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37901472,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":37028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4499","reported_seq":4555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.155461+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:57:57.155461+0000","last_peered":"2024-09-11T04:57:57.155461+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:57:57.155461+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36784448,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2097}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4480","reported_seq":4557,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.064719+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:57:59.064719+0000","last_peered":"2024-09-11T04:57:59.064719+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:57:59.064719+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680864,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4478,"num_write_kb":35822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4494","reported_seq":4550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.056529+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:57:57.056529+0000","last_peered":"2024-09-11T04:57:57.056529+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:57:57.056529+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36769520,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4333","reported_seq":6400,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.189108+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:57:57.189108+0000","last_peered":"2024-09-11T04:57:57.189108+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:57:57.189108+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4333,"log_dups_size":0,"ondisk_log_size":4333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486288,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":34655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3112}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4565","reported_seq":4632,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.169215+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:57:59.169215+0000","last_peered":"2024-09-11T04:57:59.169215+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:57:59.169215+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37386832,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":36511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4459","reported_seq":4508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:57.195599+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:57:57.195599+0000","last_peered":"2024-09-11T04:57:57.195599+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:57:57.195599+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4459,"log_dups_size":0,"ondisk_log_size":4459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36492448,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4457,"num_write_kb":35652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4502","reported_seq":4580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:57:59.095745+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:57:59.095745+0000","last_peered":"2024-09-11T04:57:59.095745+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:57:59.095745+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36852896,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4500,"num_write_kb":35997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4591","reported_seq":4654,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.410630+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:01.410630+0000","last_peered":"2024-09-11T04:58:01.410630+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:01.410630+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37565600,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4589,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":586809536,"num_objects":71637,"num_object_clones":0,"num_object_copies":286548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71664,"num_write_kb":573194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1042,"num_bytes_recovered":8314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1025662976,"data_stored":1025662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71691,"ondisk_log_size":71691,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514216,"kb_used_data":283992,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93857624,"statfs":{"total":96636764160,"available":96110206976,"internally_reserved":0,"allocated":290807808,"data_stored":290636998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":235728592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528148,"kb_used_data":285956,"kb_used_omap":1,"kb_used_meta":242174,"kb_avail":93843692,"statfs":{"total":96636764160,"available":96095940608,"internally_reserved":0,"allocated":292818944,"data_stored":292648134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":247986629},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303244,"kb_used_data":141116,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94068596,"statfs":{"total":96636764160,"available":96326242304,"internally_reserved":0,"allocated":144502784,"data_stored":144331974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":165998290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515504,"kb_used_data":289504,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93856336,"statfs":{"total":96636764160,"available":96108888064,"internally_reserved":0,"allocated":296452096,"data_stored":296281286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":231403215},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296505344,"data_stored":296505344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144400384,"data_stored":144400384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293675008,"data_stored":293675008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291082240,"data_stored":291082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:03.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:58:03.706920+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0383242 2024-09-11T04:58:03.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:58:03.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 140 20 73073 73053 4.07593 2.49219 0.0634269 0.0383242 2024-09-11T04:58:04.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 141 20 73397 73377 4.06497 2.53125 0.0432695 0.0384233 2024-09-11T04:58:05.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 142 20 73718 73698 4.05401 2.50781 0.0620292 0.038526 2024-09-11T04:58:06.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:06.672 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:06.696 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-11T04:58:05.657162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595837120,"num_objects":72739,"num_object_clones":0,"num_object_copies":290956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72766,"num_write_kb":582010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":8839168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72793,"ondisk_log_size":72793,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1893728,"kb_used_data":1019288,"kb_used_omap":14,"kb_used_meta":874353,"kb_avail":375593632,"statfs":{"total":386547056640,"available":384607879168,"internally_reserved":0,"allocated":1043750912,"data_stored":1043067672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":895338038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,14,15,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28672000,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":28000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005790"},"pg_stats":[{"pgid":"1.f","version":"31'4465","reported_seq":4507,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.083323+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:02.083323+0000","last_peered":"2024-09-11T04:58:02.083323+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:02.083323+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4465,"log_dups_size":0,"ondisk_log_size":4465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36577280,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4465,"num_write_kb":35720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4547","reported_seq":4609,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.409456+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:01.409456+0000","last_peered":"2024-09-11T04:58:01.409456+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:01.409456+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37213344,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4545,"num_write_kb":36356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4403","reported_seq":4476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:03.991031+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:03.991031+0000","last_peered":"2024-09-11T04:58:03.991031+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:03.991031+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36043344,"num_objects":4400,"num_object_clones":0,"num_object_copies":17600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4402,"num_write_kb":35206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2180}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4484","reported_seq":4547,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.376148+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:01.376148+0000","last_peered":"2024-09-11T04:58:01.376148+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:01.376148+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697248,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":35845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2202}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4516","reported_seq":4570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.207661+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:02.207661+0000","last_peered":"2024-09-11T04:58:02.207661+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:02.207661+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967584,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":36109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4594","reported_seq":4635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.253125+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:02.253125+0000","last_peered":"2024-09-11T04:58:02.253125+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:02.253125+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37634048,"num_objects":4594,"num_object_clones":0,"num_object_copies":18376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2273}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4515","reported_seq":4578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.408659+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:01.408659+0000","last_peered":"2024-09-11T04:58:01.408659+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:01.408659+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934816,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4513,"num_write_kb":36084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4632","reported_seq":4694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.385731+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:01.385731+0000","last_peered":"2024-09-11T04:58:01.385731+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:01.385731+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37901472,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":37028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4615","reported_seq":4671,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.339051+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:02.339051+0000","last_peered":"2024-09-11T04:58:02.339051+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:02.339051+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4615,"log_dups_size":0,"ondisk_log_size":4615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37734720,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":36873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4579","reported_seq":4656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.128071+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:04.128071+0000","last_peered":"2024-09-11T04:58:04.128071+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:04.128071+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37491872,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4593","reported_seq":4649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.369174+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:02.369174+0000","last_peered":"2024-09-11T04:58:02.369174+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:02.369174+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37580528,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":36708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4446","reported_seq":6641,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.384476+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:02.384476+0000","last_peered":"2024-09-11T04:58:02.384476+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:02.384476+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36411984,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":35559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":8839168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4667","reported_seq":4741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.034497+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:04.034497+0000","last_peered":"2024-09-11T04:58:04.034497+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:04.034497+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38222416,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":37327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2334}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4541","reported_seq":4591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.168193+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:02.168193+0000","last_peered":"2024-09-11T04:58:02.168193+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:02.168193+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37164192,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4539,"num_write_kb":36308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4605","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.065477+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:04.065477+0000","last_peered":"2024-09-11T04:58:04.065477+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:04.065477+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37696672,"num_objects":4602,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4603,"num_write_kb":36821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4591","reported_seq":4654,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.410630+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:01.410630+0000","last_peered":"2024-09-11T04:58:01.410630+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:01.410630+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37565600,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4589,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":595837120,"num_objects":72739,"num_object_clones":0,"num_object_copies":290956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72766,"num_write_kb":582010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":8839168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1045143552,"data_stored":1045143552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72793,"ondisk_log_size":72793,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524580,"kb_used_data":290192,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93847260,"statfs":{"total":96636764160,"available":96099594240,"internally_reserved":0,"allocated":297156608,"data_stored":296985798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":239988432},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539856,"kb_used_data":292096,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93831984,"statfs":{"total":96636764160,"available":96083951616,"internally_reserved":0,"allocated":299106304,"data_stored":298935494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313788,"kb_used_data":147496,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94058052,"statfs":{"total":96636764160,"available":96315445248,"internally_reserved":0,"allocated":151035904,"data_stored":150865094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170258130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515504,"kb_used_data":289504,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93856336,"statfs":{"total":96636764160,"available":96108888064,"internally_reserved":0,"allocated":296452096,"data_stored":296281286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":231403215},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296505344,"data_stored":296505344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151060480,"data_stored":151060480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299958272,"data_stored":299958272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297619456,"data_stored":297619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:06.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:06.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 143 20 74021 74001 4.04221 2.36719 0.0614921 0.0386419 2024-09-11T04:58:07.403 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:07.440 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-11T04:58:05.657162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595837120,"num_objects":72739,"num_object_clones":0,"num_object_copies":290956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72766,"num_write_kb":582010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":8839168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72793,"ondisk_log_size":72793,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1893728,"kb_used_data":1019288,"kb_used_omap":14,"kb_used_meta":874353,"kb_avail":375593632,"statfs":{"total":386547056640,"available":384607879168,"internally_reserved":0,"allocated":1043750912,"data_stored":1043067672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":895338038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,14,15,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28672000,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":28000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005790"},"pg_stats":[{"pgid":"1.f","version":"31'4465","reported_seq":4507,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.083323+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:02.083323+0000","last_peered":"2024-09-11T04:58:02.083323+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:02.083323+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4465,"log_dups_size":0,"ondisk_log_size":4465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36577280,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4465,"num_write_kb":35720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4547","reported_seq":4609,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.409456+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:01.409456+0000","last_peered":"2024-09-11T04:58:01.409456+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:01.409456+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37213344,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4545,"num_write_kb":36356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4403","reported_seq":4476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:03.991031+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:03.991031+0000","last_peered":"2024-09-11T04:58:03.991031+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:03.991031+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36043344,"num_objects":4400,"num_object_clones":0,"num_object_copies":17600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4402,"num_write_kb":35206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2180}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4484","reported_seq":4547,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.376148+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:01.376148+0000","last_peered":"2024-09-11T04:58:01.376148+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:01.376148+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697248,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":35845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2202}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4516","reported_seq":4570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.207661+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:02.207661+0000","last_peered":"2024-09-11T04:58:02.207661+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:02.207661+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36967584,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":36109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4594","reported_seq":4635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.253125+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:02.253125+0000","last_peered":"2024-09-11T04:58:02.253125+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:02.253125+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37634048,"num_objects":4594,"num_object_clones":0,"num_object_copies":18376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2273}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4515","reported_seq":4578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.408659+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:01.408659+0000","last_peered":"2024-09-11T04:58:01.408659+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:01.408659+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4515,"log_dups_size":0,"ondisk_log_size":4515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934816,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4513,"num_write_kb":36084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4632","reported_seq":4694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.385731+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:01.385731+0000","last_peered":"2024-09-11T04:58:01.385731+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:01.385731+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37901472,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":37028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4615","reported_seq":4671,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.339051+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:02.339051+0000","last_peered":"2024-09-11T04:58:02.339051+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:02.339051+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4615,"log_dups_size":0,"ondisk_log_size":4615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37734720,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":36873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4579","reported_seq":4656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.128071+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:04.128071+0000","last_peered":"2024-09-11T04:58:04.128071+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:04.128071+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37491872,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4593","reported_seq":4649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.369174+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:02.369174+0000","last_peered":"2024-09-11T04:58:02.369174+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:02.369174+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37580528,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":36708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4446","reported_seq":6641,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.384476+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:02.384476+0000","last_peered":"2024-09-11T04:58:02.384476+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:02.384476+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36411984,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":35559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":8839168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4667","reported_seq":4741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.034497+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:04.034497+0000","last_peered":"2024-09-11T04:58:04.034497+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:04.034497+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38222416,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":37327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2334}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4541","reported_seq":4591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:02.168193+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:02.168193+0000","last_peered":"2024-09-11T04:58:02.168193+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:02.168193+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37164192,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4539,"num_write_kb":36308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4605","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.065477+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:04.065477+0000","last_peered":"2024-09-11T04:58:04.065477+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:04.065477+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37696672,"num_objects":4602,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4603,"num_write_kb":36821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4591","reported_seq":4654,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:01.410630+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:01.410630+0000","last_peered":"2024-09-11T04:58:01.410630+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:01.410630+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37565600,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4589,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":595837120,"num_objects":72739,"num_object_clones":0,"num_object_copies":290956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72766,"num_write_kb":582010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":8839168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1045143552,"data_stored":1045143552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72793,"ondisk_log_size":72793,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524580,"kb_used_data":290192,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93847260,"statfs":{"total":96636764160,"available":96099594240,"internally_reserved":0,"allocated":297156608,"data_stored":296985798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":239988432},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539856,"kb_used_data":292096,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93831984,"statfs":{"total":96636764160,"available":96083951616,"internally_reserved":0,"allocated":299106304,"data_stored":298935494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313788,"kb_used_data":147496,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94058052,"statfs":{"total":96636764160,"available":96315445248,"internally_reserved":0,"allocated":151035904,"data_stored":150865094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170258130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515504,"kb_used_data":289504,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93856336,"statfs":{"total":96636764160,"available":96108888064,"internally_reserved":0,"allocated":296452096,"data_stored":296281286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":231403215},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296505344,"data_stored":296505344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151060480,"data_stored":151060480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299958272,"data_stored":299958272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297619456,"data_stored":297619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4465", 'reported_seq': 4507, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:02.083323+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:02.083323+0000', 'last_peered': '2024-09-11T04:58:02.083323+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:02.083323+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4465, 'log_dups_size': 0, 'ondisk_log_size': 4465, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36577280, 'num_objects': 4465, 'num_object_clones': 0, 'num_object_copies': 17860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4465, 'num_write_kb': 35720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2153}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4547", 'reported_seq': 4609, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:01.409456+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:01.409456+0000', 'last_peered': '2024-09-11T04:58:01.409456+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:01.409456+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4547, 'log_dups_size': 0, 'ondisk_log_size': 4547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37213344, 'num_objects': 4543, 'num_object_clones': 0, 'num_object_copies': 18172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4545, 'num_write_kb': 36356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4403", 'reported_seq': 4476, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:03.991031+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:03.991031+0000', 'last_peered': '2024-09-11T04:58:03.991031+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:03.991031+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4403, 'log_dups_size': 0, 'ondisk_log_size': 4403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36043344, 'num_objects': 4400, 'num_object_clones': 0, 'num_object_copies': 17600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4402, 'num_write_kb': 35206, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2180}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4484", 'reported_seq': 4547, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:01.376148+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:01.376148+0000', 'last_peered': '2024-09-11T04:58:01.376148+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:01.376148+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4484, 'log_dups_size': 0, 'ondisk_log_size': 4484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36697248, 'num_objects': 4480, 'num_object_clones': 0, 'num_object_copies': 17920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4482, 'num_write_kb': 35845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4516", 'reported_seq': 4570, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:02.207661+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:02.207661+0000', 'last_peered': '2024-09-11T04:58:02.207661+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:02.207661+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4516, 'log_dups_size': 0, 'ondisk_log_size': 4516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36967584, 'num_objects': 4513, 'num_object_clones': 0, 'num_object_copies': 18052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4514, 'num_write_kb': 36109, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4594", 'reported_seq': 4635, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:02.253125+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:02.253125+0000', 'last_peered': '2024-09-11T04:58:02.253125+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:02.253125+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4594, 'log_dups_size': 0, 'ondisk_log_size': 4594, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37634048, 'num_objects': 4594, 'num_object_clones': 0, 'num_object_copies': 18376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4594, 'num_write_kb': 36752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2273}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4515", 'reported_seq': 4578, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:01.408659+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:01.408659+0000', 'last_peered': '2024-09-11T04:58:01.408659+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:01.408659+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4515, 'log_dups_size': 0, 'ondisk_log_size': 4515, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36934816, 'num_objects': 4509, 'num_object_clones': 0, 'num_object_copies': 18036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4513, 'num_write_kb': 36084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4632", 'reported_seq': 4694, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:01.385731+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:01.385731+0000', 'last_peered': '2024-09-11T04:58:01.385731+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:01.385731+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4632, 'log_dups_size': 0, 'ondisk_log_size': 4632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37901472, 'num_objects': 4627, 'num_object_clones': 0, 'num_object_copies': 18508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4630, 'num_write_kb': 37028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4615", 'reported_seq': 4671, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:02.339051+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:02.339051+0000', 'last_peered': '2024-09-11T04:58:02.339051+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:02.339051+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4615, 'log_dups_size': 0, 'ondisk_log_size': 4615, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37734720, 'num_objects': 4607, 'num_object_clones': 0, 'num_object_copies': 18428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4612, 'num_write_kb': 36873, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2213}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4579", 'reported_seq': 4656, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:04.128071+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:04.128071+0000', 'last_peered': '2024-09-11T04:58:04.128071+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:04.128071+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4579, 'log_dups_size': 0, 'ondisk_log_size': 4579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37491872, 'num_objects': 4577, 'num_object_clones': 0, 'num_object_copies': 18308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4577, 'num_write_kb': 36614, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2259}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4593", 'reported_seq': 4649, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:02.369174+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:02.369174+0000', 'last_peered': '2024-09-11T04:58:02.369174+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:02.369174+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37580528, 'num_objects': 4588, 'num_object_clones': 0, 'num_object_copies': 18352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4590, 'num_write_kb': 36708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2265}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4446", 'reported_seq': 6641, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:02.384476+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:02.384476+0000', 'last_peered': '2024-09-11T04:58:02.384476+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:02.384476+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4446, 'log_dups_size': 0, 'ondisk_log_size': 4446, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36411984, 'num_objects': 4445, 'num_object_clones': 0, 'num_object_copies': 17780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1156, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4445, 'num_write_kb': 35559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1080, 'num_bytes_recovered': 8839168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4667", 'reported_seq': 4741, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:04.034497+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:04.034497+0000', 'last_peered': '2024-09-11T04:58:04.034497+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:04.034497+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4667, 'log_dups_size': 0, 'ondisk_log_size': 4667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38222416, 'num_objects': 4666, 'num_object_clones': 0, 'num_object_copies': 18664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4666, 'num_write_kb': 37327, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4541", 'reported_seq': 4591, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:02.168193+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:02.168193+0000', 'last_peered': '2024-09-11T04:58:02.168193+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:02.168193+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4541, 'log_dups_size': 0, 'ondisk_log_size': 4541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37164192, 'num_objects': 4537, 'num_object_clones': 0, 'num_object_copies': 18148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4539, 'num_write_kb': 36308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:07.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:07.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4605", 'reported_seq': 4683, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:04.065477+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:04.065477+0000', 'last_peered': '2024-09-11T04:58:04.065477+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:04.065477+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4605, 'log_dups_size': 0, 'ondisk_log_size': 4605, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37696672, 'num_objects': 4602, 'num_object_clones': 0, 'num_object_copies': 18408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4603, 'num_write_kb': 36821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2284}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:07.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:07.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4591", 'reported_seq': 4654, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:01.410630+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:01.410630+0000', 'last_peered': '2024-09-11T04:58:01.410630+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:01.410630+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4591, 'log_dups_size': 0, 'ondisk_log_size': 4591, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37565600, 'num_objects': 4586, 'num_object_clones': 0, 'num_object_copies': 18344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4589, 'num_write_kb': 36700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2275}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:07.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:07.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 144 20 74285 74265 4.02846 2.0625 0.0181869 0.0387404 2024-09-11T04:58:08.083 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:08.083 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35753/295684 objects degraded (12.092%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":215,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73921,"data_bytes":605520064,"bytes_used":1991180288,"bytes_avail":384555876352,"bytes_total":386547056640,"degraded_objects":35753,"degraded_total":295684,"degraded_ratio":0.12091624842737517,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":115315,"recovering_keys_per_sec":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"write_bytes_sec":2928615,"read_op_per_sec":0,"write_op_per_sec":357},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:08.107 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:08.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:08.694 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:08.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 145 20 74561 74541 4.01555 2.15625 0.0902254 0.0388968 2024-09-11T04:58:08.732 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-11T04:58:07.657895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605520064,"num_objects":73921,"num_object_clones":0,"num_object_copies":295684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73948,"num_write_kb":591466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73975,"ondisk_log_size":73975,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1944512,"kb_used_data":1038144,"kb_used_omap":14,"kb_used_meta":906289,"kb_avail":375542848,"statfs":{"total":386547056640,"available":384555876352,"internally_reserved":0,"allocated":1063059456,"data_stored":1062376216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":928040502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,16,9,16,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35160064,"num_objects":4292,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4292,"num_write_kb":34336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005694"},"pg_stats":[{"pgid":"1.f","version":"31'4577","reported_seq":4619,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.390073+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:07.390073+0000","last_peered":"2024-09-11T04:58:07.390073+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:07.390073+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4644","reported_seq":4705,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:06.347944+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:06.347944+0000","last_peered":"2024-09-11T04:58:06.347944+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:06.347944+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4644,"log_dups_size":0,"ondisk_log_size":4644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38007968,"num_objects":4640,"num_object_clones":0,"num_object_copies":18560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4642,"num_write_kb":37132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4403","reported_seq":4476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:03.991031+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:03.991031+0000","last_peered":"2024-09-11T04:58:03.991031+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:03.991031+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36043344,"num_objects":4400,"num_object_clones":0,"num_object_copies":17600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4402,"num_write_kb":35206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2180}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4594","reported_seq":4657,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:06.360277+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:06.360277+0000","last_peered":"2024-09-11T04:58:06.360277+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:06.360277+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37598368,"num_objects":4590,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4592,"num_write_kb":36725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4614","reported_seq":4668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.348248+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:07.348248+0000","last_peered":"2024-09-11T04:58:07.348248+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:07.348248+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":36893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4697","reported_seq":4738,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.318373+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:07.318373+0000","last_peered":"2024-09-11T04:58:07.318373+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:07.318373+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38477824,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4697,"num_write_kb":37576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4611","reported_seq":4674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:06.216050+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:06.216050+0000","last_peered":"2024-09-11T04:58:06.216050+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:06.216050+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37721248,"num_objects":4605,"num_object_clones":0,"num_object_copies":18420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4609,"num_write_kb":36852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2288}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4719","reported_seq":4781,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:06.438348+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:06.438348+0000","last_peered":"2024-09-11T04:58:06.438348+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:06.438348+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38614176,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4717,"num_write_kb":37724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2353}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4722","reported_seq":4778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.372415+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:07.372415+0000","last_peered":"2024-09-11T04:58:07.372415+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:07.372415+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38611264,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":37729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4579","reported_seq":4656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.128071+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:04.128071+0000","last_peered":"2024-09-11T04:58:04.128071+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:04.128071+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37491872,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4683","reported_seq":4739,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.272793+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:07.272793+0000","last_peered":"2024-09-11T04:58:07.272793+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:07.272793+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38317808,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4680,"num_write_kb":37428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4544","reported_seq":6839,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.349042+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:07.349042+0000","last_peered":"2024-09-11T04:58:07.349042+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:07.349042+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37214800,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1130,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4667","reported_seq":4741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.034497+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:04.034497+0000","last_peered":"2024-09-11T04:58:04.034497+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:04.034497+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38222416,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":37327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2334}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4630","reported_seq":4680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.285540+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:07.285540+0000","last_peered":"2024-09-11T04:58:07.285540+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:07.285540+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4605","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:04.065477+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:04.065477+0000","last_peered":"2024-09-11T04:58:04.065477+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:04.065477+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37696672,"num_objects":4602,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4603,"num_write_kb":36821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4686","reported_seq":4749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:06.288686+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:06.288686+0000","last_peered":"2024-09-11T04:58:06.288686+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:06.288686+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4686,"log_dups_size":0,"ondisk_log_size":4686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38343840,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4684,"num_write_kb":37460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":605520064,"num_objects":73921,"num_object_clones":0,"num_object_copies":295684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73948,"num_write_kb":591466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064656896,"data_stored":1064656896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73975,"ondisk_log_size":73975,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553108,"kb_used_data":296516,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93818732,"statfs":{"total":96636764160,"available":96070381568,"internally_reserved":0,"allocated":303632384,"data_stored":303461574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539856,"kb_used_data":292096,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93831984,"statfs":{"total":96636764160,"available":96083951616,"internally_reserved":0,"allocated":299106304,"data_stored":298935494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307796,"kb_used_data":153988,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94064044,"statfs":{"total":96636764160,"available":96321581056,"internally_reserved":0,"allocated":157683712,"data_stored":157512902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":157478610},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543752,"kb_used_data":295544,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93828088,"statfs":{"total":96636764160,"available":96079962112,"internally_reserved":0,"allocated":302637056,"data_stored":302466246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302964736,"data_stored":302964736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157696000,"data_stored":157696000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299958272,"data_stored":299958272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304037888,"data_stored":304037888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:09.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 146 20 74886 74866 4.00544 2.53906 0.0277333 0.0389933 2024-09-11T04:58:10.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 147 20 75203 75183 3.99503 2.47656 0.0765907 0.0390977 2024-09-11T04:58:11.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 148 20 75526 75506 3.98509 2.52344 0.0593504 0.0391963 2024-09-11T04:58:11.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:12.381 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:12.394 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-11T04:58:11.659356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612188352,"num_objects":74735,"num_object_clones":0,"num_object_copies":298940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74762,"num_write_kb":597978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74789,"ondisk_log_size":74789,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1964760,"kb_used_data":1050072,"kb_used_omap":14,"kb_used_meta":914609,"kb_avail":375522600,"statfs":{"total":386547056640,"available":384535142400,"internally_reserved":0,"allocated":1075273728,"data_stored":1074590488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":936560182},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,9,10,5,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29138944,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005504"},"pg_stats":[{"pgid":"1.f","version":"31'4577","reported_seq":4619,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.390073+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:07.390073+0000","last_peered":"2024-09-11T04:58:07.390073+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:07.390073+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4739","reported_seq":4801,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.505684+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:11.505684+0000","last_peered":"2024-09-11T04:58:11.505684+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:11.505684+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786208,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4483","reported_seq":4558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.223672+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:09.223672+0000","last_peered":"2024-09-11T04:58:09.223672+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:09.223672+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36698704,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":35846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4696","reported_seq":4759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.465216+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:11.465216+0000","last_peered":"2024-09-11T04:58:11.465216+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:11.465216+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38433952,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4694,"num_write_kb":37541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4614","reported_seq":4668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.348248+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:07.348248+0000","last_peered":"2024-09-11T04:58:07.348248+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:07.348248+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":36893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4697","reported_seq":4738,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.318373+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:07.318373+0000","last_peered":"2024-09-11T04:58:07.318373+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:07.318373+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38477824,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4697,"num_write_kb":37576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4702","reported_seq":4763,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.518821+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:11.518821+0000","last_peered":"2024-09-11T04:58:11.518821+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:11.518821+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38466720,"num_objects":4696,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":37580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2379}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4817","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.500058+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:11.500058+0000","last_peered":"2024-09-11T04:58:11.500058+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:11.500058+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39416992,"num_objects":4812,"num_object_clones":0,"num_object_copies":19248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4815,"num_write_kb":38508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4722","reported_seq":4778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.372415+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:07.372415+0000","last_peered":"2024-09-11T04:58:07.372415+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:07.372415+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38611264,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":37729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4664","reported_seq":4741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.155752+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:09.155752+0000","last_peered":"2024-09-11T04:58:09.155752+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:09.155752+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38188192,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4662,"num_write_kb":37294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2344}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4683","reported_seq":4739,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.272793+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:07.272793+0000","last_peered":"2024-09-11T04:58:07.272793+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:07.272793+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38317808,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4680,"num_write_kb":37428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4544","reported_seq":6839,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.349042+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:07.349042+0000","last_peered":"2024-09-11T04:58:07.349042+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:07.349042+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37214800,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1130,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4755","reported_seq":4827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.159410+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:09.159410+0000","last_peered":"2024-09-11T04:58:09.159410+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:09.159410+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38943312,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4754,"num_write_kb":38031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2422}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4630","reported_seq":4680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.285540+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:07.285540+0000","last_peered":"2024-09-11T04:58:07.285540+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:07.285540+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4686","reported_seq":4764,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.009001+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:09.009001+0000","last_peered":"2024-09-11T04:58:09.009001+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:09.009001+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4686,"log_dups_size":0,"ondisk_log_size":4686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360224,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4684,"num_write_kb":37469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2365}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4780","reported_seq":4843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.255098+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:11.255098+0000","last_peered":"2024-09-11T04:58:11.255098+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:11.255098+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39113888,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4778,"num_write_kb":38212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":612188352,"num_objects":74735,"num_object_clones":0,"num_object_copies":298940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74762,"num_write_kb":597978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1076961280,"data_stored":1076961280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74789,"ondisk_log_size":74789,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553108,"kb_used_data":296516,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93818732,"statfs":{"total":96636764160,"available":96070381568,"internally_reserved":0,"allocated":303632384,"data_stored":303461574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549972,"kb_used_data":298052,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93821868,"statfs":{"total":96636764160,"available":96073592832,"internally_reserved":0,"allocated":305205248,"data_stored":305034438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307796,"kb_used_data":153988,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94064044,"statfs":{"total":96636764160,"available":96321581056,"internally_reserved":0,"allocated":157683712,"data_stored":157512902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":157478610},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553884,"kb_used_data":301516,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93817956,"statfs":{"total":96636764160,"available":96069586944,"internally_reserved":0,"allocated":308752384,"data_stored":308581574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,0,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309112832,"data_stored":309112832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157696000,"data_stored":157696000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":306114560,"data_stored":306114560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304037888,"data_stored":304037888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:12.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:12.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 149 20 75775 75755 3.9714 1.94531 0.141571 0.0393153 2024-09-11T04:58:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:13.117 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:13.135 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-11T04:58:11.659356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612188352,"num_objects":74735,"num_object_clones":0,"num_object_copies":298940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74762,"num_write_kb":597978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74789,"ondisk_log_size":74789,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1964760,"kb_used_data":1050072,"kb_used_omap":14,"kb_used_meta":914609,"kb_avail":375522600,"statfs":{"total":386547056640,"available":384535142400,"internally_reserved":0,"allocated":1075273728,"data_stored":1074590488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":936560182},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,9,10,5,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29138944,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005504"},"pg_stats":[{"pgid":"1.f","version":"31'4577","reported_seq":4619,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.390073+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:07.390073+0000","last_peered":"2024-09-11T04:58:07.390073+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:07.390073+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4739","reported_seq":4801,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.505684+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:11.505684+0000","last_peered":"2024-09-11T04:58:11.505684+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:11.505684+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786208,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4483","reported_seq":4558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.223672+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:09.223672+0000","last_peered":"2024-09-11T04:58:09.223672+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:09.223672+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36698704,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":35846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4696","reported_seq":4759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.465216+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:11.465216+0000","last_peered":"2024-09-11T04:58:11.465216+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:11.465216+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38433952,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4694,"num_write_kb":37541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4614","reported_seq":4668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.348248+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:07.348248+0000","last_peered":"2024-09-11T04:58:07.348248+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:07.348248+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37770400,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":36893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4697","reported_seq":4738,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.318373+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:07.318373+0000","last_peered":"2024-09-11T04:58:07.318373+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:07.318373+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38477824,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4697,"num_write_kb":37576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4702","reported_seq":4763,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.518821+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:11.518821+0000","last_peered":"2024-09-11T04:58:11.518821+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:11.518821+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38466720,"num_objects":4696,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":37580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2379}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4817","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.500058+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:11.500058+0000","last_peered":"2024-09-11T04:58:11.500058+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:11.500058+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39416992,"num_objects":4812,"num_object_clones":0,"num_object_copies":19248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4815,"num_write_kb":38508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4722","reported_seq":4778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.372415+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:07.372415+0000","last_peered":"2024-09-11T04:58:07.372415+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:07.372415+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38611264,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":37729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4664","reported_seq":4741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.155752+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:09.155752+0000","last_peered":"2024-09-11T04:58:09.155752+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:09.155752+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38188192,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4662,"num_write_kb":37294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2344}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4683","reported_seq":4739,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.272793+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:07.272793+0000","last_peered":"2024-09-11T04:58:07.272793+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:07.272793+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38317808,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4680,"num_write_kb":37428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4544","reported_seq":6839,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.349042+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:07.349042+0000","last_peered":"2024-09-11T04:58:07.349042+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:07.349042+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37214800,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1130,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4755","reported_seq":4827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.159410+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:09.159410+0000","last_peered":"2024-09-11T04:58:09.159410+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:09.159410+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38943312,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4754,"num_write_kb":38031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2422}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4630","reported_seq":4680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:07.285540+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:07.285540+0000","last_peered":"2024-09-11T04:58:07.285540+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:07.285540+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893280,"num_objects":4626,"num_object_clones":0,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4686","reported_seq":4764,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.009001+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:09.009001+0000","last_peered":"2024-09-11T04:58:09.009001+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:09.009001+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4686,"log_dups_size":0,"ondisk_log_size":4686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360224,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4684,"num_write_kb":37469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2365}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4780","reported_seq":4843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.255098+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:11.255098+0000","last_peered":"2024-09-11T04:58:11.255098+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:11.255098+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39113888,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4778,"num_write_kb":38212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":612188352,"num_objects":74735,"num_object_clones":0,"num_object_copies":298940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74762,"num_write_kb":597978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9248768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1076961280,"data_stored":1076961280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74789,"ondisk_log_size":74789,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553108,"kb_used_data":296516,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93818732,"statfs":{"total":96636764160,"available":96070381568,"internally_reserved":0,"allocated":303632384,"data_stored":303461574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549972,"kb_used_data":298052,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93821868,"statfs":{"total":96636764160,"available":96073592832,"internally_reserved":0,"allocated":305205248,"data_stored":305034438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307796,"kb_used_data":153988,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94064044,"statfs":{"total":96636764160,"available":96321581056,"internally_reserved":0,"allocated":157683712,"data_stored":157512902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":157478610},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553884,"kb_used_data":301516,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93817956,"statfs":{"total":96636764160,"available":96069586944,"internally_reserved":0,"allocated":308752384,"data_stored":308581574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,0,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309112832,"data_stored":309112832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157696000,"data_stored":157696000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":306114560,"data_stored":306114560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304037888,"data_stored":304037888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:13.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:13.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4577", 'reported_seq': 4619, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:07.390073+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:07.390073+0000', 'last_peered': '2024-09-11T04:58:07.390073+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:07.390073+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4577, 'log_dups_size': 0, 'ondisk_log_size': 4577, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37494784, 'num_objects': 4577, 'num_object_clones': 0, 'num_object_copies': 18308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4577, 'num_write_kb': 36616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2265}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:13.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4739", 'reported_seq': 4801, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:11.505684+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:11.505684+0000', 'last_peered': '2024-09-11T04:58:11.505684+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:11.505684+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4739, 'log_dups_size': 0, 'ondisk_log_size': 4739, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38786208, 'num_objects': 4735, 'num_object_clones': 0, 'num_object_copies': 18940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4737, 'num_write_kb': 37892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2456}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4483", 'reported_seq': 4558, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:09.223672+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:09.223672+0000', 'last_peered': '2024-09-11T04:58:09.223672+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:09.223672+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4483, 'log_dups_size': 0, 'ondisk_log_size': 4483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36698704, 'num_objects': 4480, 'num_object_clones': 0, 'num_object_copies': 17920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4482, 'num_write_kb': 35846, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2260}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4696", 'reported_seq': 4759, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:11.465216+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:11.465216+0000', 'last_peered': '2024-09-11T04:58:11.465216+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:11.465216+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4696, 'log_dups_size': 0, 'ondisk_log_size': 4696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38433952, 'num_objects': 4692, 'num_object_clones': 0, 'num_object_copies': 18768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4694, 'num_write_kb': 37541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4614", 'reported_seq': 4668, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:07.348248+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:07.348248+0000', 'last_peered': '2024-09-11T04:58:07.348248+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:07.348248+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4614, 'log_dups_size': 0, 'ondisk_log_size': 4614, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37770400, 'num_objects': 4611, 'num_object_clones': 0, 'num_object_copies': 18444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4612, 'num_write_kb': 36893, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2361}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:13.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4697", 'reported_seq': 4738, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:07.318373+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:07.318373+0000', 'last_peered': '2024-09-11T04:58:07.318373+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:07.318373+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4697, 'log_dups_size': 0, 'ondisk_log_size': 4697, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38477824, 'num_objects': 4697, 'num_object_clones': 0, 'num_object_copies': 18788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4697, 'num_write_kb': 37576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2376}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4702", 'reported_seq': 4763, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:11.518821+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:11.518821+0000', 'last_peered': '2024-09-11T04:58:11.518821+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:11.518821+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4702, 'log_dups_size': 0, 'ondisk_log_size': 4702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38466720, 'num_objects': 4696, 'num_object_clones': 0, 'num_object_copies': 18784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4700, 'num_write_kb': 37580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2379}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4817", 'reported_seq': 4879, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:11.500058+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:11.500058+0000', 'last_peered': '2024-09-11T04:58:11.500058+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:11.500058+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4817, 'log_dups_size': 0, 'ondisk_log_size': 4817, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39416992, 'num_objects': 4812, 'num_object_clones': 0, 'num_object_copies': 19248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4815, 'num_write_kb': 38508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4722", 'reported_seq': 4778, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:07.372415+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:07.372415+0000', 'last_peered': '2024-09-11T04:58:07.372415+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:07.372415+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4722, 'log_dups_size': 0, 'ondisk_log_size': 4722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38611264, 'num_objects': 4714, 'num_object_clones': 0, 'num_object_copies': 18856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4719, 'num_write_kb': 37729, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2320}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4664", 'reported_seq': 4741, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:09.155752+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:09.155752+0000', 'last_peered': '2024-09-11T04:58:09.155752+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:09.155752+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4664, 'log_dups_size': 0, 'ondisk_log_size': 4664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38188192, 'num_objects': 4662, 'num_object_clones': 0, 'num_object_copies': 18648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4662, 'num_write_kb': 37294, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2344}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4683", 'reported_seq': 4739, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:07.272793+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:07.272793+0000', 'last_peered': '2024-09-11T04:58:07.272793+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:07.272793+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4683, 'log_dups_size': 0, 'ondisk_log_size': 4683, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38317808, 'num_objects': 4678, 'num_object_clones': 0, 'num_object_copies': 18712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4680, 'num_write_kb': 37428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2355}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:13.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4544", 'reported_seq': 6839, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:07.349042+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:07.349042+0000', 'last_peered': '2024-09-11T04:58:07.349042+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:07.349042+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4544, 'log_dups_size': 0, 'ondisk_log_size': 4544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37214800, 'num_objects': 4543, 'num_object_clones': 0, 'num_object_copies': 18172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4543, 'num_write_kb': 36343, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1130, 'num_bytes_recovered': 9248768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3437}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4755", 'reported_seq': 4827, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:09.159410+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:09.159410+0000', 'last_peered': '2024-09-11T04:58:09.159410+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:09.159410+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4755, 'log_dups_size': 0, 'ondisk_log_size': 4755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38943312, 'num_objects': 4754, 'num_object_clones': 0, 'num_object_copies': 19016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4754, 'num_write_kb': 38031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2422}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4630", 'reported_seq': 4680, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:07.285540+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:07.285540+0000', 'last_peered': '2024-09-11T04:58:07.285540+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:07.285540+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4630, 'log_dups_size': 0, 'ondisk_log_size': 4630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37893280, 'num_objects': 4626, 'num_object_clones': 0, 'num_object_copies': 18504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4628, 'num_write_kb': 37020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4686", 'reported_seq': 4764, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:09.009001+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:09.009001+0000', 'last_peered': '2024-09-11T04:58:09.009001+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:09.009001+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4686, 'log_dups_size': 0, 'ondisk_log_size': 4686, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38360224, 'num_objects': 4683, 'num_object_clones': 0, 'num_object_copies': 18732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4684, 'num_write_kb': 37469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2365}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:13.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4780", 'reported_seq': 4843, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:11.255098+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:11.255098+0000', 'last_peered': '2024-09-11T04:58:11.255098+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:11.255098+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4780, 'log_dups_size': 0, 'ondisk_log_size': 4780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39113888, 'num_objects': 4775, 'num_object_clones': 0, 'num_object_copies': 19100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4778, 'num_write_kb': 38212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:13.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:13.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 150 20 76053 76033 3.9594 2.17188 0.102823 0.0394486 2024-09-11T04:58:13.916 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:13.916 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35753/298940 objects degraded (11.960%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74735,"data_bytes":612188352,"bytes_used":2011914240,"bytes_avail":384535142400,"bytes_total":386547056640,"degraded_objects":35753,"degraded_total":298940,"degraded_ratio":0.11959925068575634,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":77788,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"write_bytes_sec":2427131,"read_op_per_sec":0,"write_op_per_sec":296},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:13.949 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:13.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:14.559 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:14.559 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:14.583 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-11T04:58:13.660054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617726144,"num_objects":75411,"num_object_clones":0,"num_object_copies":301644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75438,"num_write_kb":603386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":9674752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75465,"ondisk_log_size":75465,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1968508,"kb_used_data":1062140,"kb_used_omap":14,"kb_used_meta":906289,"kb_avail":375518852,"statfs":{"total":386547056640,"available":384531304448,"internally_reserved":0,"allocated":1087631360,"data_stored":1086948120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":928040502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,7,11,14,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30916608,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3774,"num_write_kb":30192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":1359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005476"},"pg_stats":[{"pgid":"1.f","version":"31'4668","reported_seq":4709,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:12.358906+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:12.358906+0000","last_peered":"2024-09-11T04:58:12.358906+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:12.358906+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38240256,"num_objects":4668,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4668,"num_write_kb":37344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4739","reported_seq":4801,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.505684+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:11.505684+0000","last_peered":"2024-09-11T04:58:11.505684+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:11.505684+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786208,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4483","reported_seq":4558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.223672+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:09.223672+0000","last_peered":"2024-09-11T04:58:09.223672+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:09.223672+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36698704,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":35846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4696","reported_seq":4759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.465216+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:11.465216+0000","last_peered":"2024-09-11T04:58:11.465216+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:11.465216+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38433952,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4694,"num_write_kb":37541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4716","reported_seq":4770,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:12.277200+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:12.277200+0000","last_peered":"2024-09-11T04:58:12.277200+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:12.277200+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38605984,"num_objects":4713,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4714,"num_write_kb":37709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4795","reported_seq":4836,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:12.355864+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:12.355864+0000","last_peered":"2024-09-11T04:58:12.355864+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:12.355864+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39280640,"num_objects":4795,"num_object_clones":0,"num_object_copies":19180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4795,"num_write_kb":38360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4702","reported_seq":4763,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.518821+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:11.518821+0000","last_peered":"2024-09-11T04:58:11.518821+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:11.518821+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38466720,"num_objects":4696,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":37580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2379}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4817","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.500058+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:11.500058+0000","last_peered":"2024-09-11T04:58:11.500058+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:11.500058+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39416992,"num_objects":4812,"num_object_clones":0,"num_object_copies":19248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4815,"num_write_kb":38508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4811","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:12.202702+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:12.202702+0000","last_peered":"2024-09-11T04:58:12.202702+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:12.202702+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4811,"log_dups_size":0,"ondisk_log_size":4811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39340352,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4664","reported_seq":4741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.155752+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:09.155752+0000","last_peered":"2024-09-11T04:58:09.155752+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:09.155752+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38188192,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4662,"num_write_kb":37294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2344}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4772","reported_seq":4828,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:12.395761+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:12.395761+0000","last_peered":"2024-09-11T04:58:12.395761+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:12.395761+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39046896,"num_objects":4767,"num_object_clones":0,"num_object_copies":19068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4769,"num_write_kb":38140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4653","reported_seq":7051,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:12.433474+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:12.433474+0000","last_peered":"2024-09-11T04:58:12.433474+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:12.433474+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38107728,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4652,"num_write_kb":37215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":9674752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3598}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4755","reported_seq":4827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.159410+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:09.159410+0000","last_peered":"2024-09-11T04:58:09.159410+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:09.159410+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38943312,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4754,"num_write_kb":38031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2422}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4728","reported_seq":4778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:12.298931+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:12.298931+0000","last_peered":"2024-09-11T04:58:12.298931+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:12.298931+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38696096,"num_objects":4724,"num_object_clones":0,"num_object_copies":18896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4726,"num_write_kb":37804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4686","reported_seq":4764,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:09.009001+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:09.009001+0000","last_peered":"2024-09-11T04:58:09.009001+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:09.009001+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4686,"log_dups_size":0,"ondisk_log_size":4686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360224,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4684,"num_write_kb":37469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2365}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4780","reported_seq":4843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:11.255098+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:11.255098+0000","last_peered":"2024-09-11T04:58:11.255098+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:11.255098+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39113888,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4778,"num_write_kb":38212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":617726144,"num_objects":75411,"num_object_clones":0,"num_object_copies":301644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75438,"num_write_kb":603386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":9674752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089417216,"data_stored":1089417216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75465,"ondisk_log_size":75465,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546584,"kb_used_data":302472,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93825256,"statfs":{"total":96636764160,"available":96077062144,"internally_reserved":0,"allocated":309731328,"data_stored":309560518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549972,"kb_used_data":298052,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93821868,"statfs":{"total":96636764160,"available":96073592832,"internally_reserved":0,"allocated":305205248,"data_stored":305034438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318068,"kb_used_data":160100,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94053772,"statfs":{"total":96636764160,"available":96311062528,"internally_reserved":0,"allocated":163942400,"data_stored":163771590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":161738450},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553884,"kb_used_data":301516,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93817956,"statfs":{"total":96636764160,"available":96069586944,"internally_reserved":0,"allocated":308752384,"data_stored":308581574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,0,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309112832,"data_stored":309112832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164061184,"data_stored":164061184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":306114560,"data_stored":306114560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310128640,"data_stored":310128640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:14.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 151 20 76336 76316 3.94783 2.21094 0.0316068 0.0395693 2024-09-11T04:58:15.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 152 20 76649 76629 3.93794 2.44531 0.035226 0.039663 2024-09-11T04:58:16.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 153 20 76969 76949 3.92854 2.5 0.0754592 0.0397627 2024-09-11T04:58:17.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:17.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 154 20 77276 77256 3.91861 2.39844 0.0705235 0.0398561 2024-09-11T04:58:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:18.251 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-11T04:58:17.661743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":629932224,"num_objects":76901,"num_object_clones":0,"num_object_copies":307604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76928,"num_write_kb":615306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76955,"ondisk_log_size":76955,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2009760,"kb_used_data":1086748,"kb_used_omap":14,"kb_used_meta":922929,"kb_avail":375477600,"statfs":{"total":386547056640,"available":384489062400,"internally_reserved":0,"allocated":1112829952,"data_stored":1112146712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":945079862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,5,5,11,16,23,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34095104,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4162,"num_write_kb":33296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004580"},"pg_stats":[{"pgid":"1.f","version":"31'4773","reported_seq":4815,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.315118+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:17.315118+0000","last_peered":"2024-09-11T04:58:17.315118+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:17.315118+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39100416,"num_objects":4773,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4773,"num_write_kb":38184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4831","reported_seq":4893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.530104+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:16.530104+0000","last_peered":"2024-09-11T04:58:16.530104+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:16.530104+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39539872,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4595","reported_seq":4665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:14.158313+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:14.158313+0000","last_peered":"2024-09-11T04:58:14.158313+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:14.158313+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37616208,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4779","reported_seq":4842,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.502570+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:16.502570+0000","last_peered":"2024-09-11T04:58:16.502570+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:16.502570+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39113888,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4810","reported_seq":4864,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.410969+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:17.410969+0000","last_peered":"2024-09-11T04:58:17.410969+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:17.410969+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376032,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2557}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4882","reported_seq":4923,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.436936+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:17.436936+0000","last_peered":"2024-09-11T04:58:17.436936+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:17.436936+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39993344,"num_objects":4882,"num_object_clones":0,"num_object_copies":19528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4882,"num_write_kb":39056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4804","reported_seq":4866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.560825+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:16.560825+0000","last_peered":"2024-09-11T04:58:16.560825+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:16.560825+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302304,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4802,"num_write_kb":38396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2481}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4920","reported_seq":4982,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.480217+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:16.480217+0000","last_peered":"2024-09-11T04:58:16.480217+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:16.480217+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40260768,"num_objects":4915,"num_object_clones":0,"num_object_copies":19660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4907","reported_seq":4963,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.362960+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:17.362960+0000","last_peered":"2024-09-11T04:58:17.362960+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:17.362960+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40126784,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4904,"num_write_kb":39209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4757","reported_seq":4834,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:14.182010+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:14.182010+0000","last_peered":"2024-09-11T04:58:14.182010+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:14.182010+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4757,"log_dups_size":0,"ondisk_log_size":4757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38950048,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2437}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4858","reported_seq":4914,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.398229+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:17.398229+0000","last_peered":"2024-09-11T04:58:17.398229+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:17.398229+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39751408,"num_objects":4853,"num_object_clones":0,"num_object_copies":19412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":38828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4734","reported_seq":7239,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.418878+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:17.418878+0000","last_peered":"2024-09-11T04:58:17.418878+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:17.418878+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38771280,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1235,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4836","reported_seq":4910,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:14.101974+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:14.101974+0000","last_peered":"2024-09-11T04:58:14.101974+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:14.101974+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39606864,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":38679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4807","reported_seq":4857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.415204+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:17.415204+0000","last_peered":"2024-09-11T04:58:17.415204+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:17.415204+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343264,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4805,"num_write_kb":38436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4779","reported_seq":4857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:14.213137+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:14.213137+0000","last_peered":"2024-09-11T04:58:14.213137+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:14.213137+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39122080,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4883","reported_seq":4946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.430657+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:16.430657+0000","last_peered":"2024-09-11T04:58:16.430657+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:16.430657+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39957664,"num_objects":4878,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":39036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":629932224,"num_objects":76901,"num_object_clones":0,"num_object_copies":307604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76928,"num_write_kb":615306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1113776128,"data_stored":1113776128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76955,"ondisk_log_size":76955,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556416,"kb_used_data":308144,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93815424,"statfs":{"total":96636764160,"available":96066994176,"internally_reserved":0,"allocated":315539456,"data_stored":315368646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561120,"kb_used_data":305040,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93810720,"statfs":{"total":96636764160,"available":96062177280,"internally_reserved":0,"allocated":312360960,"data_stored":312190150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,5,2,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328304,"kb_used_data":166176,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94043536,"statfs":{"total":96636764160,"available":96300580864,"internally_reserved":0,"allocated":170164224,"data_stored":169993414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":165998290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563920,"kb_used_data":307388,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93807920,"statfs":{"total":96636764160,"available":96059310080,"internally_reserved":0,"allocated":314765312,"data_stored":314594502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,8,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315084800,"data_stored":315084800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170340352,"data_stored":170340352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312135680,"data_stored":312135680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316215296,"data_stored":316215296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:18.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:18.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 155 20 77553 77533 3.90729 2.16406 0.0559241 0.0399704 2024-09-11T04:58:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:18.867 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:18.893 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-11T04:58:17.661743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":629932224,"num_objects":76901,"num_object_clones":0,"num_object_copies":307604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76928,"num_write_kb":615306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76955,"ondisk_log_size":76955,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2009760,"kb_used_data":1086748,"kb_used_omap":14,"kb_used_meta":922929,"kb_avail":375477600,"statfs":{"total":386547056640,"available":384489062400,"internally_reserved":0,"allocated":1112829952,"data_stored":1112146712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":945079862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,5,5,11,16,23,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34095104,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4162,"num_write_kb":33296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004580"},"pg_stats":[{"pgid":"1.f","version":"31'4773","reported_seq":4815,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.315118+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:17.315118+0000","last_peered":"2024-09-11T04:58:17.315118+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:17.315118+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39100416,"num_objects":4773,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4773,"num_write_kb":38184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4831","reported_seq":4893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.530104+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:16.530104+0000","last_peered":"2024-09-11T04:58:16.530104+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:16.530104+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39539872,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4595","reported_seq":4665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:14.158313+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:14.158313+0000","last_peered":"2024-09-11T04:58:14.158313+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:14.158313+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37616208,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4779","reported_seq":4842,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.502570+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:16.502570+0000","last_peered":"2024-09-11T04:58:16.502570+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:16.502570+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39113888,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4810","reported_seq":4864,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.410969+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:17.410969+0000","last_peered":"2024-09-11T04:58:17.410969+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:17.410969+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376032,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2557}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4882","reported_seq":4923,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.436936+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:17.436936+0000","last_peered":"2024-09-11T04:58:17.436936+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:17.436936+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39993344,"num_objects":4882,"num_object_clones":0,"num_object_copies":19528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4882,"num_write_kb":39056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4804","reported_seq":4866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.560825+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:16.560825+0000","last_peered":"2024-09-11T04:58:16.560825+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:16.560825+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302304,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4802,"num_write_kb":38396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2481}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4920","reported_seq":4982,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.480217+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:16.480217+0000","last_peered":"2024-09-11T04:58:16.480217+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:16.480217+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40260768,"num_objects":4915,"num_object_clones":0,"num_object_copies":19660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4907","reported_seq":4963,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.362960+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:17.362960+0000","last_peered":"2024-09-11T04:58:17.362960+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:17.362960+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40126784,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4904,"num_write_kb":39209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4757","reported_seq":4834,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:14.182010+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:14.182010+0000","last_peered":"2024-09-11T04:58:14.182010+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:14.182010+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4757,"log_dups_size":0,"ondisk_log_size":4757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38950048,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":38038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2437}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4858","reported_seq":4914,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.398229+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:17.398229+0000","last_peered":"2024-09-11T04:58:17.398229+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:17.398229+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39751408,"num_objects":4853,"num_object_clones":0,"num_object_copies":19412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":38828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4734","reported_seq":7239,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.418878+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:17.418878+0000","last_peered":"2024-09-11T04:58:17.418878+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:17.418878+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38771280,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1235,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4836","reported_seq":4910,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:14.101974+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:14.101974+0000","last_peered":"2024-09-11T04:58:14.101974+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:14.101974+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39606864,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":38679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4807","reported_seq":4857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.415204+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:17.415204+0000","last_peered":"2024-09-11T04:58:17.415204+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:17.415204+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343264,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4805,"num_write_kb":38436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4779","reported_seq":4857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:14.213137+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:14.213137+0000","last_peered":"2024-09-11T04:58:14.213137+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:14.213137+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39122080,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4883","reported_seq":4946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.430657+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:16.430657+0000","last_peered":"2024-09-11T04:58:16.430657+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:16.430657+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39957664,"num_objects":4878,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":39036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":629932224,"num_objects":76901,"num_object_clones":0,"num_object_copies":307604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76928,"num_write_kb":615306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1113776128,"data_stored":1113776128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76955,"ondisk_log_size":76955,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556416,"kb_used_data":308144,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93815424,"statfs":{"total":96636764160,"available":96066994176,"internally_reserved":0,"allocated":315539456,"data_stored":315368646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561120,"kb_used_data":305040,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93810720,"statfs":{"total":96636764160,"available":96062177280,"internally_reserved":0,"allocated":312360960,"data_stored":312190150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,5,2,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328304,"kb_used_data":166176,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94043536,"statfs":{"total":96636764160,"available":96300580864,"internally_reserved":0,"allocated":170164224,"data_stored":169993414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":165998290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563920,"kb_used_data":307388,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93807920,"statfs":{"total":96636764160,"available":96059310080,"internally_reserved":0,"allocated":314765312,"data_stored":314594502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,8,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315084800,"data_stored":315084800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170340352,"data_stored":170340352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312135680,"data_stored":312135680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316215296,"data_stored":316215296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:18.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:18.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4773", 'reported_seq': 4815, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:17.315118+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:17.315118+0000', 'last_peered': '2024-09-11T04:58:17.315118+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:17.315118+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4773, 'log_dups_size': 0, 'ondisk_log_size': 4773, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39100416, 'num_objects': 4773, 'num_object_clones': 0, 'num_object_copies': 19092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4773, 'num_write_kb': 38184, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2461}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:18.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:18.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4831", 'reported_seq': 4893, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:16.530104+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:16.530104+0000', 'last_peered': '2024-09-11T04:58:16.530104+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:16.530104+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4831, 'log_dups_size': 0, 'ondisk_log_size': 4831, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39539872, 'num_objects': 4827, 'num_object_clones': 0, 'num_object_copies': 19308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4829, 'num_write_kb': 38628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2548}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4595", 'reported_seq': 4665, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:14.158313+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:14.158313+0000', 'last_peered': '2024-09-11T04:58:14.158313+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:14.158313+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4595, 'log_dups_size': 0, 'ondisk_log_size': 4595, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37616208, 'num_objects': 4592, 'num_object_clones': 0, 'num_object_copies': 18368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4594, 'num_write_kb': 36742, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2372}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4779", 'reported_seq': 4842, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:16.502570+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:16.502570+0000', 'last_peered': '2024-09-11T04:58:16.502570+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:16.502570+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4779, 'log_dups_size': 0, 'ondisk_log_size': 4779, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39113888, 'num_objects': 4775, 'num_object_clones': 0, 'num_object_copies': 19100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4777, 'num_write_kb': 38205, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2497}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4810", 'reported_seq': 4864, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:17.410969+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:17.410969+0000', 'last_peered': '2024-09-11T04:58:17.410969+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:17.410969+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4810, 'log_dups_size': 0, 'ondisk_log_size': 4810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39376032, 'num_objects': 4807, 'num_object_clones': 0, 'num_object_copies': 19228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4808, 'num_write_kb': 38461, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2557}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4882", 'reported_seq': 4923, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:17.436936+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:17.436936+0000', 'last_peered': '2024-09-11T04:58:17.436936+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:17.436936+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4882, 'log_dups_size': 0, 'ondisk_log_size': 4882, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39993344, 'num_objects': 4882, 'num_object_clones': 0, 'num_object_copies': 19528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4882, 'num_write_kb': 39056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2561}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4804", 'reported_seq': 4866, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:16.560825+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:16.560825+0000', 'last_peered': '2024-09-11T04:58:16.560825+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:16.560825+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4804, 'log_dups_size': 0, 'ondisk_log_size': 4804, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39302304, 'num_objects': 4798, 'num_object_clones': 0, 'num_object_copies': 19192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4802, 'num_write_kb': 38396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2481}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:18.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4920", 'reported_seq': 4982, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:16.480217+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:16.480217+0000', 'last_peered': '2024-09-11T04:58:16.480217+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:16.480217+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4920, 'log_dups_size': 0, 'ondisk_log_size': 4920, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40260768, 'num_objects': 4915, 'num_object_clones': 0, 'num_object_copies': 19660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4918, 'num_write_kb': 39332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2554}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4907", 'reported_seq': 4963, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:17.362960+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:17.362960+0000', 'last_peered': '2024-09-11T04:58:17.362960+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:17.362960+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4907, 'log_dups_size': 0, 'ondisk_log_size': 4907, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40126784, 'num_objects': 4899, 'num_object_clones': 0, 'num_object_copies': 19596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4904, 'num_write_kb': 39209, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2505}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4757", 'reported_seq': 4834, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:14.182010+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:14.182010+0000', 'last_peered': '2024-09-11T04:58:14.182010+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:14.182010+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4757, 'log_dups_size': 0, 'ondisk_log_size': 4757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38950048, 'num_objects': 4755, 'num_object_clones': 0, 'num_object_copies': 19020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4755, 'num_write_kb': 38038, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2437}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4858", 'reported_seq': 4914, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:17.398229+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:17.398229+0000', 'last_peered': '2024-09-11T04:58:17.398229+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:17.398229+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4858, 'log_dups_size': 0, 'ondisk_log_size': 4858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39751408, 'num_objects': 4853, 'num_object_clones': 0, 'num_object_copies': 19412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4855, 'num_write_kb': 38828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2530}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4734", 'reported_seq': 7239, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:17.418878+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:17.418878+0000', 'last_peered': '2024-09-11T04:58:17.418878+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:17.418878+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4734, 'log_dups_size': 0, 'ondisk_log_size': 4734, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38771280, 'num_objects': 4733, 'num_object_clones': 0, 'num_object_copies': 18932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4733, 'num_write_kb': 37863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1235, 'num_bytes_recovered': 10100736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:18.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:18.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4836", 'reported_seq': 4910, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:14.101974+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:14.101974+0000', 'last_peered': '2024-09-11T04:58:14.101974+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:14.101974+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4836, 'log_dups_size': 0, 'ondisk_log_size': 4836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39606864, 'num_objects': 4835, 'num_object_clones': 0, 'num_object_copies': 19340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4835, 'num_write_kb': 38679, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:18.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:18.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4807", 'reported_seq': 4857, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:17.415204+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:17.415204+0000', 'last_peered': '2024-09-11T04:58:17.415204+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:17.415204+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4807, 'log_dups_size': 0, 'ondisk_log_size': 4807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39343264, 'num_objects': 4803, 'num_object_clones': 0, 'num_object_copies': 19212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4805, 'num_write_kb': 38436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2516}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:18.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:18.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4779", 'reported_seq': 4857, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:14.213137+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:14.213137+0000', 'last_peered': '2024-09-11T04:58:14.213137+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:14.213137+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4779, 'log_dups_size': 0, 'ondisk_log_size': 4779, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39122080, 'num_objects': 4776, 'num_object_clones': 0, 'num_object_copies': 19104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4777, 'num_write_kb': 38213, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2458}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:18.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:18.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4883", 'reported_seq': 4946, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:16.430657+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:16.430657+0000', 'last_peered': '2024-09-11T04:58:16.430657+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:16.430657+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4883, 'log_dups_size': 0, 'ondisk_log_size': 4883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39957664, 'num_objects': 4878, 'num_object_clones': 0, 'num_object_copies': 19512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4881, 'num_write_kb': 39036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2567}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:18.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:19.631 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:19.631 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35649/307604 objects degraded (11.589%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":226,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76901,"data_bytes":629932224,"bytes_used":2057994240,"bytes_avail":384489062400,"bytes_total":386547056640,"degraded_objects":35649,"degraded_total":307604,"degraded_ratio":0.11589251115070026,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105090,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":1261568,"num_keys_recovered":0,"write_bytes_sec":2840174,"read_op_per_sec":0,"write_op_per_sec":346},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:19.650 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:19.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:19.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 156 20 77830 77810 3.89611 2.16406 0.0636038 0.0400903 2024-09-11T04:58:20.289 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:20.289 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:20.309 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-11T04:58:19.662644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632856768,"num_objects":77258,"num_object_clones":0,"num_object_copies":309032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77285,"num_write_kb":618162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77312,"ondisk_log_size":77312,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2003248,"kb_used_data":1092716,"kb_used_omap":14,"kb_used_meta":910449,"kb_avail":375484112,"statfs":{"total":386547056640,"available":384495730688,"internally_reserved":0,"allocated":1118941184,"data_stored":1118257944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":932300342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,2,1,11,18,22,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27336704,"num_objects":3337,"num_object_clones":0,"num_object_copies":13348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3337,"num_write_kb":26696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004749"},"pg_stats":[{"pgid":"1.f","version":"31'4773","reported_seq":4815,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.315118+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:17.315118+0000","last_peered":"2024-09-11T04:58:17.315118+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:17.315118+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39100416,"num_objects":4773,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4773,"num_write_kb":38184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4831","reported_seq":4893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.530104+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:16.530104+0000","last_peered":"2024-09-11T04:58:16.530104+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:16.530104+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39539872,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4684","reported_seq":4759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.254188+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:19.254188+0000","last_peered":"2024-09-11T04:58:19.254188+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:19.254188+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38345296,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4779","reported_seq":4842,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.502570+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:16.502570+0000","last_peered":"2024-09-11T04:58:16.502570+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:16.502570+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39113888,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4810","reported_seq":4864,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.410969+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:17.410969+0000","last_peered":"2024-09-11T04:58:17.410969+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:17.410969+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376032,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":38461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2557}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4882","reported_seq":4923,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.436936+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:17.436936+0000","last_peered":"2024-09-11T04:58:17.436936+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:17.436936+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39993344,"num_objects":4882,"num_object_clones":0,"num_object_copies":19528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4882,"num_write_kb":39056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4804","reported_seq":4866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.560825+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:16.560825+0000","last_peered":"2024-09-11T04:58:16.560825+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:16.560825+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302304,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4802,"num_write_kb":38396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2481}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4920","reported_seq":4982,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.480217+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:16.480217+0000","last_peered":"2024-09-11T04:58:16.480217+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:16.480217+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40260768,"num_objects":4915,"num_object_clones":0,"num_object_copies":19660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4907","reported_seq":4963,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.362960+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:17.362960+0000","last_peered":"2024-09-11T04:58:17.362960+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:17.362960+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40126784,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4904,"num_write_kb":39209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4863","reported_seq":4940,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.233448+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:19.233448+0000","last_peered":"2024-09-11T04:58:19.233448+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:19.233448+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39818400,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4861,"num_write_kb":38886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2543}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4858","reported_seq":4914,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.398229+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:17.398229+0000","last_peered":"2024-09-11T04:58:17.398229+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:17.398229+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39751408,"num_objects":4853,"num_object_clones":0,"num_object_copies":19412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":38828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4734","reported_seq":7239,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.418878+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:17.418878+0000","last_peered":"2024-09-11T04:58:17.418878+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:17.418878+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38771280,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1235,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4917","reported_seq":4991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.156835+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:19.156835+0000","last_peered":"2024-09-11T04:58:19.156835+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:19.156835+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40270416,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":39327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2584}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4807","reported_seq":4857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:17.415204+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:17.415204+0000","last_peered":"2024-09-11T04:58:17.415204+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:17.415204+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343264,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4805,"num_write_kb":38436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4860","reported_seq":4938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.124425+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:19.124425+0000","last_peered":"2024-09-11T04:58:19.124425+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:19.124425+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39785632,"num_objects":4857,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4883","reported_seq":4946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:16.430657+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:16.430657+0000","last_peered":"2024-09-11T04:58:16.430657+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:16.430657+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39957664,"num_objects":4878,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":39036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":632856768,"num_objects":77258,"num_object_clones":0,"num_object_copies":309032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77285,"num_write_kb":618162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":10100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1119924224,"data_stored":1119924224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77312,"ondisk_log_size":77312,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556416,"kb_used_data":308144,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93815424,"statfs":{"total":96636764160,"available":96066994176,"internally_reserved":0,"allocated":315539456,"data_stored":315368646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554608,"kb_used_data":311008,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93817232,"statfs":{"total":96636764160,"available":96068845568,"internally_reserved":0,"allocated":318472192,"data_stored":318301382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328304,"kb_used_data":166176,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94043536,"statfs":{"total":96636764160,"available":96300580864,"internally_reserved":0,"allocated":170164224,"data_stored":169993414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":165998290},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563920,"kb_used_data":307388,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93807920,"statfs":{"total":96636764160,"available":96059310080,"internally_reserved":0,"allocated":314765312,"data_stored":314594502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,8,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315084800,"data_stored":315084800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170340352,"data_stored":170340352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318283776,"data_stored":318283776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316215296,"data_stored":316215296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:20.710 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 157 20 78115 78095 3.88548 2.22656 0.0677252 0.0401984 2024-09-11T04:58:21.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 158 20 78434 78414 3.87665 2.49219 0.099991 0.0402913 2024-09-11T04:58:22.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 159 20 78741 78721 3.86736 2.39844 0.0651236 0.0403877 2024-09-11T04:58:23.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:23.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:58:23.708650+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0404872 2024-09-11T04:58:23.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:58:23.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 160 20 79039 79019 3.85774 2.32812 0.0767746 0.0404872 2024-09-11T04:58:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:23.966 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:23.990 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-11T04:58:23.664546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642121920,"num_objects":78389,"num_object_clones":0,"num_object_copies":313556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78416,"num_write_kb":627210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78443,"ondisk_log_size":78443,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2017116,"kb_used_data":1110748,"kb_used_omap":14,"kb_used_meta":906289,"kb_avail":375470244,"statfs":{"total":386547056640,"available":384481529856,"internally_reserved":0,"allocated":1137405952,"data_stored":1136722712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":928040502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,7,9,11,21,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29933568,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":1302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005190"},"pg_stats":[{"pgid":"1.f","version":"31'4864","reported_seq":4906,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.414856+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:22.414856+0000","last_peered":"2024-09-11T04:58:22.414856+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:22.414856+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39845888,"num_objects":4864,"num_object_clones":0,"num_object_copies":19456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4864,"num_write_kb":38912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2552}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4923","reported_seq":4985,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.466032+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:21.466032+0000","last_peered":"2024-09-11T04:58:21.466032+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:21.466032+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40293536,"num_objects":4919,"num_object_clones":0,"num_object_copies":19676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4921,"num_write_kb":39364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2640}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4684","reported_seq":4759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.254188+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:19.254188+0000","last_peered":"2024-09-11T04:58:19.254188+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:19.254188+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38345296,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4871","reported_seq":4934,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.519130+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:21.519130+0000","last_peered":"2024-09-11T04:58:21.519130+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:21.519130+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39867552,"num_objects":4867,"num_object_clones":0,"num_object_copies":19468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4869,"num_write_kb":38941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4888","reported_seq":4942,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.364088+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:22.364088+0000","last_peered":"2024-09-11T04:58:22.364088+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:22.364088+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40015008,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":39085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4972","reported_seq":5013,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.512974+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:22.512974+0000","last_peered":"2024-09-11T04:58:22.512974+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:22.512974+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40730624,"num_objects":4972,"num_object_clones":0,"num_object_copies":19888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":39776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4916","reported_seq":4978,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.557744+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:21.557744+0000","last_peered":"2024-09-11T04:58:21.557744+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:21.557744+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219808,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4914,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2593}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5015","reported_seq":5077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.605631+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:21.605631+0000","last_peered":"2024-09-11T04:58:21.605631+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:21.605631+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41039008,"num_objects":5010,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":40092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5007","reported_seq":5063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.439029+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:22.439029+0000","last_peered":"2024-09-11T04:58:22.439029+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:22.439029+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40945984,"num_objects":4999,"num_object_clones":0,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4863","reported_seq":4940,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.233448+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:19.233448+0000","last_peered":"2024-09-11T04:58:19.233448+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:19.233448+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39818400,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4861,"num_write_kb":38886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2543}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4960","reported_seq":5016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.460761+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:22.460761+0000","last_peered":"2024-09-11T04:58:22.460761+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:22.460761+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40586992,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4957,"num_write_kb":39644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4819","reported_seq":7434,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.490363+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:22.490363+0000","last_peered":"2024-09-11T04:58:22.490363+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:22.490363+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39467600,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1290,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3871}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4917","reported_seq":4991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.156835+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:19.156835+0000","last_peered":"2024-09-11T04:58:19.156835+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:19.156835+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40270416,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":39327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2584}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4914","reported_seq":4964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.450210+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:22.450210+0000","last_peered":"2024-09-11T04:58:22.450210+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:22.450210+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219808,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4860","reported_seq":4938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.124425+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:19.124425+0000","last_peered":"2024-09-11T04:58:19.124425+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:19.124425+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39785632,"num_objects":4857,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4970","reported_seq":5032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.577977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:21.577977+0000","last_peered":"2024-09-11T04:58:21.577977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:21.577977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40670368,"num_objects":4965,"num_object_clones":0,"num_object_copies":19860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642121920,"num_objects":78389,"num_object_clones":0,"num_object_copies":313556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78416,"num_write_kb":627210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138323456,"data_stored":1138323456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78443,"ondisk_log_size":78443,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566508,"kb_used_data":314076,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93805332,"statfs":{"total":96636764160,"available":96056659968,"internally_reserved":0,"allocated":321613824,"data_stored":321443014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554608,"kb_used_data":311008,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93817232,"statfs":{"total":96636764160,"available":96068845568,"internally_reserved":0,"allocated":318472192,"data_stored":318301382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338540,"kb_used_data":172252,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94033300,"statfs":{"total":96636764160,"available":96290099200,"internally_reserved":0,"allocated":176386048,"data_stored":176215238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170258130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557460,"kb_used_data":313412,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93814380,"statfs":{"total":96636764160,"available":96065925120,"internally_reserved":0,"allocated":320933888,"data_stored":320763078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249884367},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321142784,"data_stored":321142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176594944,"data_stored":176594944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318283776,"data_stored":318283776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322301952,"data_stored":322301952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:23.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:24.644 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:24.670 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-11T04:58:23.664546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642121920,"num_objects":78389,"num_object_clones":0,"num_object_copies":313556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78416,"num_write_kb":627210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78443,"ondisk_log_size":78443,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2017116,"kb_used_data":1110748,"kb_used_omap":14,"kb_used_meta":906289,"kb_avail":375470244,"statfs":{"total":386547056640,"available":384481529856,"internally_reserved":0,"allocated":1137405952,"data_stored":1136722712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":928040502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,7,9,11,21,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29933568,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":1302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005190"},"pg_stats":[{"pgid":"1.f","version":"31'4864","reported_seq":4906,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.414856+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:22.414856+0000","last_peered":"2024-09-11T04:58:22.414856+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:22.414856+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39845888,"num_objects":4864,"num_object_clones":0,"num_object_copies":19456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4864,"num_write_kb":38912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2552}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4923","reported_seq":4985,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.466032+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:21.466032+0000","last_peered":"2024-09-11T04:58:21.466032+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:21.466032+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40293536,"num_objects":4919,"num_object_clones":0,"num_object_copies":19676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4921,"num_write_kb":39364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2640}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4684","reported_seq":4759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.254188+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:19.254188+0000","last_peered":"2024-09-11T04:58:19.254188+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:19.254188+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38345296,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4871","reported_seq":4934,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.519130+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:21.519130+0000","last_peered":"2024-09-11T04:58:21.519130+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:21.519130+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39867552,"num_objects":4867,"num_object_clones":0,"num_object_copies":19468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4869,"num_write_kb":38941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4888","reported_seq":4942,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.364088+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:22.364088+0000","last_peered":"2024-09-11T04:58:22.364088+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:22.364088+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40015008,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":39085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4972","reported_seq":5013,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.512974+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:22.512974+0000","last_peered":"2024-09-11T04:58:22.512974+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:22.512974+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40730624,"num_objects":4972,"num_object_clones":0,"num_object_copies":19888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":39776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4916","reported_seq":4978,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.557744+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:21.557744+0000","last_peered":"2024-09-11T04:58:21.557744+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:21.557744+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219808,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4914,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2593}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5015","reported_seq":5077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.605631+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:21.605631+0000","last_peered":"2024-09-11T04:58:21.605631+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:21.605631+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41039008,"num_objects":5010,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":40092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5007","reported_seq":5063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.439029+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:22.439029+0000","last_peered":"2024-09-11T04:58:22.439029+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:22.439029+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40945984,"num_objects":4999,"num_object_clones":0,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4863","reported_seq":4940,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.233448+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:19.233448+0000","last_peered":"2024-09-11T04:58:19.233448+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:19.233448+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39818400,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4861,"num_write_kb":38886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2543}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4960","reported_seq":5016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.460761+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:22.460761+0000","last_peered":"2024-09-11T04:58:22.460761+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:22.460761+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40586992,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4957,"num_write_kb":39644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4819","reported_seq":7434,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.490363+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:22.490363+0000","last_peered":"2024-09-11T04:58:22.490363+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:22.490363+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39467600,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1290,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3871}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4917","reported_seq":4991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.156835+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:19.156835+0000","last_peered":"2024-09-11T04:58:19.156835+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:19.156835+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40270416,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":39327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2584}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4914","reported_seq":4964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.450210+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:22.450210+0000","last_peered":"2024-09-11T04:58:22.450210+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:22.450210+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219808,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4860","reported_seq":4938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:19.124425+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:19.124425+0000","last_peered":"2024-09-11T04:58:19.124425+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:19.124425+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39785632,"num_objects":4857,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4970","reported_seq":5032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.577977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:21.577977+0000","last_peered":"2024-09-11T04:58:21.577977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:21.577977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40670368,"num_objects":4965,"num_object_clones":0,"num_object_copies":19860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642121920,"num_objects":78389,"num_object_clones":0,"num_object_copies":313556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78416,"num_write_kb":627210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138323456,"data_stored":1138323456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78443,"ondisk_log_size":78443,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566508,"kb_used_data":314076,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93805332,"statfs":{"total":96636764160,"available":96056659968,"internally_reserved":0,"allocated":321613824,"data_stored":321443014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554608,"kb_used_data":311008,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93817232,"statfs":{"total":96636764160,"available":96068845568,"internally_reserved":0,"allocated":318472192,"data_stored":318301382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338540,"kb_used_data":172252,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94033300,"statfs":{"total":96636764160,"available":96290099200,"internally_reserved":0,"allocated":176386048,"data_stored":176215238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170258130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557460,"kb_used_data":313412,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93814380,"statfs":{"total":96636764160,"available":96065925120,"internally_reserved":0,"allocated":320933888,"data_stored":320763078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249884367},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321142784,"data_stored":321142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176594944,"data_stored":176594944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318283776,"data_stored":318283776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322301952,"data_stored":322301952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:24.671 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:24.671 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4864", 'reported_seq': 4906, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:22.414856+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:22.414856+0000', 'last_peered': '2024-09-11T04:58:22.414856+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:22.414856+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4864, 'log_dups_size': 0, 'ondisk_log_size': 4864, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39845888, 'num_objects': 4864, 'num_object_clones': 0, 'num_object_copies': 19456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4864, 'num_write_kb': 38912, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2552}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:24.671 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:24.671 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4923", 'reported_seq': 4985, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:21.466032+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:21.466032+0000', 'last_peered': '2024-09-11T04:58:21.466032+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:21.466032+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4923, 'log_dups_size': 0, 'ondisk_log_size': 4923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40293536, 'num_objects': 4919, 'num_object_clones': 0, 'num_object_copies': 19676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4921, 'num_write_kb': 39364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2640}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:24.671 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4684", 'reported_seq': 4759, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:19.254188+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:19.254188+0000', 'last_peered': '2024-09-11T04:58:19.254188+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:19.254188+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4684, 'log_dups_size': 0, 'ondisk_log_size': 4684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38345296, 'num_objects': 4681, 'num_object_clones': 0, 'num_object_copies': 18724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4683, 'num_write_kb': 37454, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2461}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4871", 'reported_seq': 4934, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:21.519130+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:21.519130+0000', 'last_peered': '2024-09-11T04:58:21.519130+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:21.519130+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4871, 'log_dups_size': 0, 'ondisk_log_size': 4871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39867552, 'num_objects': 4867, 'num_object_clones': 0, 'num_object_copies': 19468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4869, 'num_write_kb': 38941, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2589}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4888", 'reported_seq': 4942, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:22.364088+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:22.364088+0000', 'last_peered': '2024-09-11T04:58:22.364088+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:22.364088+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4888, 'log_dups_size': 0, 'ondisk_log_size': 4888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40015008, 'num_objects': 4885, 'num_object_clones': 0, 'num_object_copies': 19540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4886, 'num_write_kb': 39085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2635}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4972", 'reported_seq': 5013, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:22.512974+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:22.512974+0000', 'last_peered': '2024-09-11T04:58:22.512974+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:22.512974+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4972, 'log_dups_size': 0, 'ondisk_log_size': 4972, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40730624, 'num_objects': 4972, 'num_object_clones': 0, 'num_object_copies': 19888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4972, 'num_write_kb': 39776, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2651}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4916", 'reported_seq': 4978, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:21.557744+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:21.557744+0000', 'last_peered': '2024-09-11T04:58:21.557744+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:21.557744+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4916, 'log_dups_size': 0, 'ondisk_log_size': 4916, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40219808, 'num_objects': 4910, 'num_object_clones': 0, 'num_object_copies': 19640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4914, 'num_write_kb': 39292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:24.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5015", 'reported_seq': 5077, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:21.605631+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:21.605631+0000', 'last_peered': '2024-09-11T04:58:21.605631+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:21.605631+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5015, 'log_dups_size': 0, 'ondisk_log_size': 5015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41039008, 'num_objects': 5010, 'num_object_clones': 0, 'num_object_copies': 20040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5013, 'num_write_kb': 40092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5007", 'reported_seq': 5063, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:22.439029+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:22.439029+0000', 'last_peered': '2024-09-11T04:58:22.439029+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:22.439029+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5007, 'log_dups_size': 0, 'ondisk_log_size': 5007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40945984, 'num_objects': 4999, 'num_object_clones': 0, 'num_object_copies': 19996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5004, 'num_write_kb': 40009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2605}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4863", 'reported_seq': 4940, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:19.233448+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:19.233448+0000', 'last_peered': '2024-09-11T04:58:19.233448+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:19.233448+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4863, 'log_dups_size': 0, 'ondisk_log_size': 4863, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39818400, 'num_objects': 4861, 'num_object_clones': 0, 'num_object_copies': 19444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4861, 'num_write_kb': 38886, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2543}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4960", 'reported_seq': 5016, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:22.460761+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:22.460761+0000', 'last_peered': '2024-09-11T04:58:22.460761+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:22.460761+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4960, 'log_dups_size': 0, 'ondisk_log_size': 4960, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40586992, 'num_objects': 4955, 'num_object_clones': 0, 'num_object_copies': 19820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4957, 'num_write_kb': 39644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2632}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4819", 'reported_seq': 7434, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:22.490363+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:22.490363+0000', 'last_peered': '2024-09-11T04:58:22.490363+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:22.490363+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4819, 'log_dups_size': 0, 'ondisk_log_size': 4819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39467600, 'num_objects': 4818, 'num_object_clones': 0, 'num_object_copies': 19272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 947, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4818, 'num_write_kb': 38543, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1290, 'num_bytes_recovered': 10551296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:24.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4917", 'reported_seq': 4991, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:19.156835+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:19.156835+0000', 'last_peered': '2024-09-11T04:58:19.156835+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:19.156835+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4917, 'log_dups_size': 0, 'ondisk_log_size': 4917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40270416, 'num_objects': 4916, 'num_object_clones': 0, 'num_object_copies': 19664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4916, 'num_write_kb': 39327, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2584}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:24.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:24.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4914", 'reported_seq': 4964, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:22.450210+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:22.450210+0000', 'last_peered': '2024-09-11T04:58:22.450210+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:22.450210+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4914, 'log_dups_size': 0, 'ondisk_log_size': 4914, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40219808, 'num_objects': 4910, 'num_object_clones': 0, 'num_object_copies': 19640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4912, 'num_write_kb': 39292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2623}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:24.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:24.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4860", 'reported_seq': 4938, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:19.124425+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:19.124425+0000', 'last_peered': '2024-09-11T04:58:19.124425+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:19.124425+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4860, 'log_dups_size': 0, 'ondisk_log_size': 4860, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39785632, 'num_objects': 4857, 'num_object_clones': 0, 'num_object_copies': 19428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4858, 'num_write_kb': 38861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2539}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:24.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:24.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4970", 'reported_seq': 5032, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:21.577977+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:21.577977+0000', 'last_peered': '2024-09-11T04:58:21.577977+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:21.577977+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4970, 'log_dups_size': 0, 'ondisk_log_size': 4970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40670368, 'num_objects': 4965, 'num_object_clones': 0, 'num_object_copies': 19860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4968, 'num_write_kb': 39732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:24.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:24.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 161 20 79315 79295 3.84717 2.15625 0.0707941 0.0406011 2024-09-11T04:58:25.464 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:25.465 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35594/313556 objects degraded (11.352%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":232,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78389,"data_bytes":642121920,"bytes_used":2065526784,"bytes_avail":384481529856,"bytes_total":386547056640,"degraded_objects":35594,"degraded_total":313556,"degraded_ratio":0.11351720266874178,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":108497,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":1302528,"num_keys_recovered":0,"write_bytes_sec":2493385,"read_op_per_sec":0,"write_op_per_sec":304},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:25.499 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:25.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:25.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 162 20 79591 79571 3.83673 2.15625 0.0590066 0.0407117 2024-09-11T04:58:26.137 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:26.137 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:26.162 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-11T04:58:25.665442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644817088,"num_objects":78718,"num_object_clones":0,"num_object_copies":314872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78745,"num_write_kb":629842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78772,"ondisk_log_size":78772,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2027148,"kb_used_data":1116620,"kb_used_omap":14,"kb_used_meta":910449,"kb_avail":375460212,"statfs":{"total":386547056640,"available":384471257088,"internally_reserved":0,"allocated":1143418880,"data_stored":1142735640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":932300342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,8,11,6,16,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27090944,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":26456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005387"},"pg_stats":[{"pgid":"1.f","version":"31'4864","reported_seq":4906,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.414856+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:22.414856+0000","last_peered":"2024-09-11T04:58:22.414856+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:22.414856+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39845888,"num_objects":4864,"num_object_clones":0,"num_object_copies":19456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4864,"num_write_kb":38912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2552}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4923","reported_seq":4985,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.466032+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:21.466032+0000","last_peered":"2024-09-11T04:58:21.466032+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:21.466032+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40293536,"num_objects":4919,"num_object_clones":0,"num_object_copies":19676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4921,"num_write_kb":39364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2640}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4776","reported_seq":4850,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:24.219430+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:24.219430+0000","last_peered":"2024-09-11T04:58:24.219430+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:24.219430+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39098960,"num_objects":4773,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":38190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2553}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4871","reported_seq":4934,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.519130+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:21.519130+0000","last_peered":"2024-09-11T04:58:21.519130+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:21.519130+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39867552,"num_objects":4867,"num_object_clones":0,"num_object_copies":19468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4869,"num_write_kb":38941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4888","reported_seq":4942,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.364088+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:22.364088+0000","last_peered":"2024-09-11T04:58:22.364088+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:22.364088+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40015008,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":39085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4972","reported_seq":5013,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.512974+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:22.512974+0000","last_peered":"2024-09-11T04:58:22.512974+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:22.512974+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40730624,"num_objects":4972,"num_object_clones":0,"num_object_copies":19888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":39776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4916","reported_seq":4978,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.557744+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:21.557744+0000","last_peered":"2024-09-11T04:58:21.557744+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:21.557744+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219808,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4914,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2593}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5015","reported_seq":5077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.605631+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:21.605631+0000","last_peered":"2024-09-11T04:58:21.605631+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:21.605631+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41039008,"num_objects":5010,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":40092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5007","reported_seq":5063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.439029+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:22.439029+0000","last_peered":"2024-09-11T04:58:22.439029+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:22.439029+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40945984,"num_objects":4999,"num_object_clones":0,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4972","reported_seq":5049,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:24.188975+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:24.188975+0000","last_peered":"2024-09-11T04:58:24.188975+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:24.188975+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40711328,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2652}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4960","reported_seq":5016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.460761+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:22.460761+0000","last_peered":"2024-09-11T04:58:22.460761+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:22.460761+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40586992,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4957,"num_write_kb":39644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4819","reported_seq":7434,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.490363+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:22.490363+0000","last_peered":"2024-09-11T04:58:22.490363+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:22.490363+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39467600,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1290,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3871}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4988","reported_seq":5062,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:24.142005+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:24.142005+0000","last_peered":"2024-09-11T04:58:24.142005+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:24.142005+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40852048,"num_objects":4987,"num_object_clones":0,"num_object_copies":19948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4987,"num_write_kb":39895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2655}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4914","reported_seq":4964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:22.450210+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:22.450210+0000","last_peered":"2024-09-11T04:58:22.450210+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:22.450210+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219808,"num_objects":4910,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4917","reported_seq":4995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:24.232620+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:24.232620+0000","last_peered":"2024-09-11T04:58:24.232620+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:24.232620+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40252576,"num_objects":4914,"num_object_clones":0,"num_object_copies":19656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4915,"num_write_kb":39317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4970","reported_seq":5032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:21.577977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:21.577977+0000","last_peered":"2024-09-11T04:58:21.577977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:21.577977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40670368,"num_objects":4965,"num_object_clones":0,"num_object_copies":19860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":644817088,"num_objects":78718,"num_object_clones":0,"num_object_copies":314872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78745,"num_write_kb":629842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":10551296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1144455168,"data_stored":1144455168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78772,"ondisk_log_size":78772,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566508,"kb_used_data":314076,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93805332,"statfs":{"total":96636764160,"available":96056659968,"internally_reserved":0,"allocated":321613824,"data_stored":321443014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564640,"kb_used_data":316880,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93807200,"statfs":{"total":96636764160,"available":96058572800,"internally_reserved":0,"allocated":324485120,"data_stored":324314310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338540,"kb_used_data":172252,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94033300,"statfs":{"total":96636764160,"available":96290099200,"internally_reserved":0,"allocated":176386048,"data_stored":176215238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":170258130},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557460,"kb_used_data":313412,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93814380,"statfs":{"total":96636764160,"available":96065925120,"internally_reserved":0,"allocated":320933888,"data_stored":320763078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249884367},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321142784,"data_stored":321142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176594944,"data_stored":176594944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":324415488,"data_stored":324415488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322301952,"data_stored":322301952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:26.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 163 20 79874 79854 3.82675 2.21094 0.0636951 0.0408172 2024-09-11T04:58:27.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 164 20 80179 80159 3.81795 2.38281 0.0682376 0.0409092 2024-09-11T04:58:28.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 165 20 80488 80468 3.80944 2.41406 0.056281 0.0410046 2024-09-11T04:58:29.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:29.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 166 20 80778 80758 3.80014 2.26562 0.0762252 0.0411029 2024-09-11T04:58:29.862 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:29.892 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-11T04:58:29.667278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656949440,"num_objects":80199,"num_object_clones":0,"num_object_copies":320796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80226,"num_write_kb":641690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80253,"ondisk_log_size":80253,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2068592,"kb_used_data":1140016,"kb_used_omap":14,"kb_used_meta":928497,"kb_avail":375418768,"statfs":{"total":386547056640,"available":384428818432,"internally_reserved":0,"allocated":1167376384,"data_stored":1166693144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":950781494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,7,9,15,9,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27017216,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3298,"num_write_kb":26384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005535"},"pg_stats":[{"pgid":"1.f","version":"31'4946","reported_seq":4988,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.515991+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:27.515991+0000","last_peered":"2024-09-11T04:58:27.515991+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:27.515991+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40517632,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":39568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5010","reported_seq":5072,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.522795+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:26.522795+0000","last_peered":"2024-09-11T04:58:26.522795+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:26.522795+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006240,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4863","reported_seq":4938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.273925+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:29.273925+0000","last_peered":"2024-09-11T04:58:29.273925+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:29.273925+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811664,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4972","reported_seq":5035,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.580027+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:26.580027+0000","last_peered":"2024-09-11T04:58:26.580027+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:26.580027+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40694944,"num_objects":4968,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4982","reported_seq":5036,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.565686+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:27.565686+0000","last_peered":"2024-09-11T04:58:27.565686+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:27.565686+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":39837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5070","reported_seq":5111,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.519447+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:27.519447+0000","last_peered":"2024-09-11T04:58:27.519447+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:27.519447+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41533440,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5031","reported_seq":5094,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.541036+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:26.541036+0000","last_peered":"2024-09-11T04:58:26.541036+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:26.541036+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41161888,"num_objects":5025,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":40212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2708}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5090","reported_seq":5152,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.463634+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:26.463634+0000","last_peered":"2024-09-11T04:58:26.463634+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:26.463634+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41653408,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5088,"num_write_kb":40692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5091","reported_seq":5147,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.406031+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:27.406031+0000","last_peered":"2024-09-11T04:58:27.406031+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:27.406031+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41634112,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5088,"num_write_kb":40681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5055","reported_seq":5132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.112601+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:29.112601+0000","last_peered":"2024-09-11T04:58:29.112601+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:29.112601+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41391264,"num_objects":5053,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5053,"num_write_kb":40422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2735}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5062","reported_seq":5118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.624089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:27.624089+0000","last_peered":"2024-09-11T04:58:27.624089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:27.624089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5062,"log_dups_size":0,"ondisk_log_size":5062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41422576,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":40460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4889","reported_seq":7636,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.604557+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:27.604557+0000","last_peered":"2024-09-11T04:58:27.604557+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:27.604557+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40041040,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1356,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5092","reported_seq":5166,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.143274+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:29.143274+0000","last_peered":"2024-09-11T04:58:29.143274+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:29.143274+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41704016,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":40727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5018","reported_seq":5068,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.484035+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:27.484035+0000","last_peered":"2024-09-11T04:58:27.484035+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:27.484035+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41071776,"num_objects":5014,"num_object_clones":0,"num_object_copies":20056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5016,"num_write_kb":40124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2727}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5020","reported_seq":5097,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.269507+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:29.269507+0000","last_peered":"2024-09-11T04:58:29.269507+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:29.269507+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5062","reported_seq":5125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.607032+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:26.607032+0000","last_peered":"2024-09-11T04:58:26.607032+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:26.607032+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5062,"log_dups_size":0,"ondisk_log_size":5062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41424032,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":40468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":656949440,"num_objects":80199,"num_object_clones":0,"num_object_copies":320796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80226,"num_write_kb":641690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1168531456,"data_stored":1168531456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80253,"ondisk_log_size":80253,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576412,"kb_used_data":319820,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93795428,"statfs":{"total":96636764160,"available":96046518272,"internally_reserved":0,"allocated":327495680,"data_stored":327324870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,5,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574648,"kb_used_data":322728,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93797192,"statfs":{"total":96636764160,"available":96048324608,"internally_reserved":0,"allocated":330473472,"data_stored":330302662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,6,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350112,"kb_used_data":178256,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":94021728,"statfs":{"total":96636764160,"available":96278249472,"internally_reserved":0,"allocated":182534144,"data_stored":182363334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":175959762},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567420,"kb_used_data":319212,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93804420,"statfs":{"total":96636764160,"available":96055726080,"internally_reserved":0,"allocated":326873088,"data_stored":326702278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327049216,"data_stored":327049216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182775808,"data_stored":182775808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330399744,"data_stored":330399744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328306688,"data_stored":328306688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:29.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:30.492 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:30.492 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:30.519 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-11T04:58:29.667278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656949440,"num_objects":80199,"num_object_clones":0,"num_object_copies":320796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80226,"num_write_kb":641690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80253,"ondisk_log_size":80253,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2068592,"kb_used_data":1140016,"kb_used_omap":14,"kb_used_meta":928497,"kb_avail":375418768,"statfs":{"total":386547056640,"available":384428818432,"internally_reserved":0,"allocated":1167376384,"data_stored":1166693144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":950781494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,7,9,15,9,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27017216,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3298,"num_write_kb":26384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005535"},"pg_stats":[{"pgid":"1.f","version":"31'4946","reported_seq":4988,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.515991+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:27.515991+0000","last_peered":"2024-09-11T04:58:27.515991+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:27.515991+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40517632,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":39568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5010","reported_seq":5072,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.522795+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:26.522795+0000","last_peered":"2024-09-11T04:58:26.522795+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:26.522795+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006240,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4863","reported_seq":4938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.273925+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:29.273925+0000","last_peered":"2024-09-11T04:58:29.273925+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:29.273925+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811664,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4972","reported_seq":5035,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.580027+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:26.580027+0000","last_peered":"2024-09-11T04:58:26.580027+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:26.580027+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40694944,"num_objects":4968,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4982","reported_seq":5036,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.565686+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:27.565686+0000","last_peered":"2024-09-11T04:58:27.565686+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:27.565686+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":39837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5070","reported_seq":5111,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.519447+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:27.519447+0000","last_peered":"2024-09-11T04:58:27.519447+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:27.519447+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41533440,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5031","reported_seq":5094,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.541036+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:26.541036+0000","last_peered":"2024-09-11T04:58:26.541036+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:26.541036+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41161888,"num_objects":5025,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":40212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2708}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5090","reported_seq":5152,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.463634+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:26.463634+0000","last_peered":"2024-09-11T04:58:26.463634+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:26.463634+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41653408,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5088,"num_write_kb":40692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5091","reported_seq":5147,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.406031+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:27.406031+0000","last_peered":"2024-09-11T04:58:27.406031+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:27.406031+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41634112,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5088,"num_write_kb":40681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5055","reported_seq":5132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.112601+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:29.112601+0000","last_peered":"2024-09-11T04:58:29.112601+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:29.112601+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41391264,"num_objects":5053,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5053,"num_write_kb":40422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2735}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5062","reported_seq":5118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.624089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:27.624089+0000","last_peered":"2024-09-11T04:58:27.624089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:27.624089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5062,"log_dups_size":0,"ondisk_log_size":5062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41422576,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":40460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4889","reported_seq":7636,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.604557+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:27.604557+0000","last_peered":"2024-09-11T04:58:27.604557+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:27.604557+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40041040,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1356,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5092","reported_seq":5166,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.143274+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:29.143274+0000","last_peered":"2024-09-11T04:58:29.143274+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:29.143274+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41704016,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":40727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5018","reported_seq":5068,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.484035+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:27.484035+0000","last_peered":"2024-09-11T04:58:27.484035+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:27.484035+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41071776,"num_objects":5014,"num_object_clones":0,"num_object_copies":20056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5016,"num_write_kb":40124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2727}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5020","reported_seq":5097,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.269507+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:29.269507+0000","last_peered":"2024-09-11T04:58:29.269507+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:29.269507+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5062","reported_seq":5125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.607032+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:26.607032+0000","last_peered":"2024-09-11T04:58:26.607032+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:26.607032+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5062,"log_dups_size":0,"ondisk_log_size":5062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41424032,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":40468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":656949440,"num_objects":80199,"num_object_clones":0,"num_object_copies":320796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80226,"num_write_kb":641690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1168531456,"data_stored":1168531456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80253,"ondisk_log_size":80253,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576412,"kb_used_data":319820,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93795428,"statfs":{"total":96636764160,"available":96046518272,"internally_reserved":0,"allocated":327495680,"data_stored":327324870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,5,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574648,"kb_used_data":322728,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93797192,"statfs":{"total":96636764160,"available":96048324608,"internally_reserved":0,"allocated":330473472,"data_stored":330302662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,6,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350112,"kb_used_data":178256,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":94021728,"statfs":{"total":96636764160,"available":96278249472,"internally_reserved":0,"allocated":182534144,"data_stored":182363334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":175959762},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567420,"kb_used_data":319212,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93804420,"statfs":{"total":96636764160,"available":96055726080,"internally_reserved":0,"allocated":326873088,"data_stored":326702278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327049216,"data_stored":327049216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182775808,"data_stored":182775808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330399744,"data_stored":330399744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328306688,"data_stored":328306688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:30.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:30.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4946", 'reported_seq': 4988, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:27.515991+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:27.515991+0000', 'last_peered': '2024-09-11T04:58:27.515991+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:27.515991+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4946, 'log_dups_size': 0, 'ondisk_log_size': 4946, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40517632, 'num_objects': 4946, 'num_object_clones': 0, 'num_object_copies': 19784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4946, 'num_write_kb': 39568, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2634}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:30.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:30.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5010", 'reported_seq': 5072, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:26.522795+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:26.522795+0000', 'last_peered': '2024-09-11T04:58:26.522795+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:26.522795+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5010, 'log_dups_size': 0, 'ondisk_log_size': 5010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41006240, 'num_objects': 5006, 'num_object_clones': 0, 'num_object_copies': 20024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5008, 'num_write_kb': 40060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2727}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:30.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:30.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4863", 'reported_seq': 4938, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:29.273925+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:29.273925+0000', 'last_peered': '2024-09-11T04:58:29.273925+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:29.273925+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4863, 'log_dups_size': 0, 'ondisk_log_size': 4863, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39811664, 'num_objects': 4860, 'num_object_clones': 0, 'num_object_copies': 19440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4862, 'num_write_kb': 38886, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2640}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:30.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:30.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4972", 'reported_seq': 5035, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:26.580027+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:26.580027+0000', 'last_peered': '2024-09-11T04:58:26.580027+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:26.580027+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4972, 'log_dups_size': 0, 'ondisk_log_size': 4972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40694944, 'num_objects': 4968, 'num_object_clones': 0, 'num_object_copies': 19872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4970, 'num_write_kb': 39749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:30.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:30.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4982", 'reported_seq': 5036, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:27.565686+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:27.565686+0000', 'last_peered': '2024-09-11T04:58:27.565686+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:27.565686+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4982, 'log_dups_size': 0, 'ondisk_log_size': 4982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40785056, 'num_objects': 4979, 'num_object_clones': 0, 'num_object_copies': 19916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4980, 'num_write_kb': 39837, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2729}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:30.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:30.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5070", 'reported_seq': 5111, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:27.519447+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:27.519447+0000', 'last_peered': '2024-09-11T04:58:27.519447+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:27.519447+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5070, 'log_dups_size': 0, 'ondisk_log_size': 5070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41533440, 'num_objects': 5070, 'num_object_clones': 0, 'num_object_copies': 20280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5070, 'num_write_kb': 40560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:30.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:30.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5031", 'reported_seq': 5094, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:26.541036+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:26.541036+0000', 'last_peered': '2024-09-11T04:58:26.541036+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:26.541036+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5031, 'log_dups_size': 0, 'ondisk_log_size': 5031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41161888, 'num_objects': 5025, 'num_object_clones': 0, 'num_object_copies': 20100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5029, 'num_write_kb': 40212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2708}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5090", 'reported_seq': 5152, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:26.463634+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:26.463634+0000', 'last_peered': '2024-09-11T04:58:26.463634+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:26.463634+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5090, 'log_dups_size': 0, 'ondisk_log_size': 5090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41653408, 'num_objects': 5085, 'num_object_clones': 0, 'num_object_copies': 20340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5088, 'num_write_kb': 40692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2724}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5091", 'reported_seq': 5147, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:27.406031+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:27.406031+0000', 'last_peered': '2024-09-11T04:58:27.406031+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:27.406031+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5091, 'log_dups_size': 0, 'ondisk_log_size': 5091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41634112, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5088, 'num_write_kb': 40681, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2689}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5055", 'reported_seq': 5132, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:29.112601+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:29.112601+0000', 'last_peered': '2024-09-11T04:58:29.112601+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:29.112601+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5055, 'log_dups_size': 0, 'ondisk_log_size': 5055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41391264, 'num_objects': 5053, 'num_object_clones': 0, 'num_object_copies': 20212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5053, 'num_write_kb': 40422, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2735}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5062", 'reported_seq': 5118, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:27.624089+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:27.624089+0000', 'last_peered': '2024-09-11T04:58:27.624089+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:27.624089+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5062, 'log_dups_size': 0, 'ondisk_log_size': 5062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41422576, 'num_objects': 5057, 'num_object_clones': 0, 'num_object_copies': 20228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5059, 'num_write_kb': 40460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2734}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:30.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4889", 'reported_seq': 7636, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:27.604557+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:27.604557+0000', 'last_peered': '2024-09-11T04:58:27.604557+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:27.604557+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4889, 'log_dups_size': 0, 'ondisk_log_size': 4889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40041040, 'num_objects': 4888, 'num_object_clones': 0, 'num_object_copies': 19552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 881, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4888, 'num_write_kb': 39103, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1356, 'num_bytes_recovered': 11091968, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:30.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:30.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5092", 'reported_seq': 5166, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:29.143274+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:29.143274+0000', 'last_peered': '2024-09-11T04:58:29.143274+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:29.143274+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5092, 'log_dups_size': 0, 'ondisk_log_size': 5092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41704016, 'num_objects': 5091, 'num_object_clones': 0, 'num_object_copies': 20364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5091, 'num_write_kb': 40727, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2759}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:30.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:30.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5018", 'reported_seq': 5068, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:27.484035+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:27.484035+0000', 'last_peered': '2024-09-11T04:58:27.484035+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:27.484035+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5018, 'log_dups_size': 0, 'ondisk_log_size': 5018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41071776, 'num_objects': 5014, 'num_object_clones': 0, 'num_object_copies': 20056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5016, 'num_write_kb': 40124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2727}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:30.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:30.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5020", 'reported_seq': 5097, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:29.269507+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:29.269507+0000', 'last_peered': '2024-09-11T04:58:29.269507+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:29.269507+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5020, 'log_dups_size': 0, 'ondisk_log_size': 5020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41096352, 'num_objects': 5017, 'num_object_clones': 0, 'num_object_copies': 20068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5018, 'num_write_kb': 40141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:30.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:30.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5062", 'reported_seq': 5125, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:26.607032+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:26.607032+0000', 'last_peered': '2024-09-11T04:58:26.607032+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:26.607032+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5062, 'log_dups_size': 0, 'ondisk_log_size': 5062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41424032, 'num_objects': 5057, 'num_object_clones': 0, 'num_object_copies': 20228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5060, 'num_write_kb': 40468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2746}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:30.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:30.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 167 20 81042 81022 3.78973 2.0625 0.062212 0.0412162 2024-09-11T04:58:31.298 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:31.298 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35528/320796 objects degraded (11.075%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":238,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80199,"data_bytes":656949440,"bytes_used":2118238208,"bytes_avail":384428818432,"bytes_total":386547056640,"degraded_objects":35528,"degraded_total":320796,"degraded_ratio":0.1107495105924014,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":82564,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":2250396,"read_op_per_sec":0,"write_op_per_sec":274},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:31.324 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:31.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:31.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 168 20 81308 81288 3.77954 2.07812 0.0483344 0.0413256 2024-09-11T04:58:31.946 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:31.946 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:31.971 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-11T04:58:31.668276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656949440,"num_objects":80199,"num_object_clones":0,"num_object_copies":320796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80226,"num_write_kb":641690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80253,"ondisk_log_size":80253,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2068592,"kb_used_data":1140016,"kb_used_omap":14,"kb_used_meta":928497,"kb_avail":375418768,"statfs":{"total":386547056640,"available":384428818432,"internally_reserved":0,"allocated":1167376384,"data_stored":1166693144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":950781494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,7,9,15,9,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24092672,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2941,"num_write_kb":23528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005632"},"pg_stats":[{"pgid":"1.f","version":"31'4946","reported_seq":4988,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.515991+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:27.515991+0000","last_peered":"2024-09-11T04:58:27.515991+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:27.515991+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40517632,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":39568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5010","reported_seq":5072,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.522795+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:26.522795+0000","last_peered":"2024-09-11T04:58:26.522795+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:26.522795+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006240,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4863","reported_seq":4938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.273925+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:29.273925+0000","last_peered":"2024-09-11T04:58:29.273925+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:29.273925+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811664,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4972","reported_seq":5035,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.580027+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:26.580027+0000","last_peered":"2024-09-11T04:58:26.580027+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:26.580027+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40694944,"num_objects":4968,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4970,"num_write_kb":39749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4982","reported_seq":5036,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.565686+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:27.565686+0000","last_peered":"2024-09-11T04:58:27.565686+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:27.565686+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785056,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":39837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5070","reported_seq":5111,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.519447+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:27.519447+0000","last_peered":"2024-09-11T04:58:27.519447+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:27.519447+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41533440,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5031","reported_seq":5094,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.541036+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:26.541036+0000","last_peered":"2024-09-11T04:58:26.541036+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:26.541036+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41161888,"num_objects":5025,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":40212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2708}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5090","reported_seq":5152,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.463634+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:26.463634+0000","last_peered":"2024-09-11T04:58:26.463634+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:26.463634+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41653408,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5088,"num_write_kb":40692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5091","reported_seq":5147,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.406031+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:27.406031+0000","last_peered":"2024-09-11T04:58:27.406031+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:27.406031+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41634112,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5088,"num_write_kb":40681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5055","reported_seq":5132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.112601+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:29.112601+0000","last_peered":"2024-09-11T04:58:29.112601+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:29.112601+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41391264,"num_objects":5053,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5053,"num_write_kb":40422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2735}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5062","reported_seq":5118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.624089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:27.624089+0000","last_peered":"2024-09-11T04:58:27.624089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:27.624089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5062,"log_dups_size":0,"ondisk_log_size":5062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41422576,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":40460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4889","reported_seq":7636,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.604557+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:27.604557+0000","last_peered":"2024-09-11T04:58:27.604557+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:27.604557+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40041040,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":39103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1356,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5092","reported_seq":5166,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.143274+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:29.143274+0000","last_peered":"2024-09-11T04:58:29.143274+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:29.143274+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41704016,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":40727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5018","reported_seq":5068,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:27.484035+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:27.484035+0000","last_peered":"2024-09-11T04:58:27.484035+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:27.484035+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41071776,"num_objects":5014,"num_object_clones":0,"num_object_copies":20056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5016,"num_write_kb":40124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2727}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5020","reported_seq":5097,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.269507+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:29.269507+0000","last_peered":"2024-09-11T04:58:29.269507+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:29.269507+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5062","reported_seq":5125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:26.607032+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:26.607032+0000","last_peered":"2024-09-11T04:58:26.607032+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:26.607032+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5062,"log_dups_size":0,"ondisk_log_size":5062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41424032,"num_objects":5057,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":40468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":656949440,"num_objects":80199,"num_object_clones":0,"num_object_copies":320796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80226,"num_write_kb":641690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":11091968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1168531456,"data_stored":1168531456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80253,"ondisk_log_size":80253,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576412,"kb_used_data":319820,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93795428,"statfs":{"total":96636764160,"available":96046518272,"internally_reserved":0,"allocated":327495680,"data_stored":327324870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,5,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574648,"kb_used_data":322728,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93797192,"statfs":{"total":96636764160,"available":96048324608,"internally_reserved":0,"allocated":330473472,"data_stored":330302662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,6,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350112,"kb_used_data":178256,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":94021728,"statfs":{"total":96636764160,"available":96278249472,"internally_reserved":0,"allocated":182534144,"data_stored":182363334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":175959762},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567420,"kb_used_data":319212,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93804420,"statfs":{"total":96636764160,"available":96055726080,"internally_reserved":0,"allocated":326873088,"data_stored":326702278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327049216,"data_stored":327049216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182775808,"data_stored":182775808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330399744,"data_stored":330399744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328306688,"data_stored":328306688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:32.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 169 20 81597 81577 3.77054 2.25781 0.0621397 0.0414244 2024-09-11T04:58:33.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 170 20 81896 81876 3.7621 2.33594 0.0474963 0.0415183 2024-09-11T04:58:34.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 171 20 82199 82179 3.75394 2.36719 0.047112 0.0416085 2024-09-11T04:58:34.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:35.578 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:35.606 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-11T04:58:33.669042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665559232,"num_objects":81250,"num_object_clones":0,"num_object_copies":325000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81277,"num_write_kb":650098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81304,"ondisk_log_size":81304,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2101092,"kb_used_data":1158628,"kb_used_omap":14,"kb_used_meta":942385,"kb_avail":375386268,"statfs":{"total":386547056640,"available":384395538432,"internally_reserved":0,"allocated":1186435072,"data_stored":1185751832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":965002806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,9,7,13,18,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28786688,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3514,"num_write_kb":28112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005359"},"pg_stats":[{"pgid":"1.f","version":"31'5039","reported_seq":5081,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.581185+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:32.581185+0000","last_peered":"2024-09-11T04:58:32.581185+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:32.581185+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41279488,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5039,"num_write_kb":40312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5105","reported_seq":5167,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.564057+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:31.564057+0000","last_peered":"2024-09-11T04:58:31.564057+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:31.564057+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41784480,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5103,"num_write_kb":40820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4863","reported_seq":4938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.273925+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:29.273925+0000","last_peered":"2024-09-11T04:58:29.273925+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:29.273925+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811664,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5068","reported_seq":5128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.528335+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:31.528335+0000","last_peered":"2024-09-11T04:58:31.528335+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:31.528335+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41481376,"num_objects":5064,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":40517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5074","reported_seq":5128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.567115+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:32.567115+0000","last_peered":"2024-09-11T04:58:32.567115+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:32.567115+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41538720,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5072,"num_write_kb":40573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5146","reported_seq":5187,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.640057+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:32.640057+0000","last_peered":"2024-09-11T04:58:32.640057+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:32.640057+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5146,"log_dups_size":0,"ondisk_log_size":5146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42156032,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5146,"num_write_kb":41168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5129","reported_seq":5192,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.557383+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:31.557383+0000","last_peered":"2024-09-11T04:58:31.557383+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:31.557383+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41964704,"num_objects":5123,"num_object_clones":0,"num_object_copies":20492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5127,"num_write_kb":40996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5176","reported_seq":5238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.595343+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:31.595343+0000","last_peered":"2024-09-11T04:58:31.595343+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:31.595343+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5176,"log_dups_size":0,"ondisk_log_size":5176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42357920,"num_objects":5171,"num_object_clones":0,"num_object_copies":20684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5174,"num_write_kb":41380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2810}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5182","reported_seq":5237,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.667131+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:32.667131+0000","last_peered":"2024-09-11T04:58:32.667131+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:32.667131+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5182,"log_dups_size":0,"ondisk_log_size":5182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42379584,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5179,"num_write_kb":41409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5055","reported_seq":5132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.112601+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:29.112601+0000","last_peered":"2024-09-11T04:58:29.112601+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:29.112601+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41391264,"num_objects":5053,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5053,"num_write_kb":40422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2735}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5139","reported_seq":5195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.365453+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:32.365453+0000","last_peered":"2024-09-11T04:58:32.365453+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:32.365453+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42053360,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4961","reported_seq":7818,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.679714+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:32.679714+0000","last_peered":"2024-09-11T04:58:32.679714+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:32.679714+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4961,"log_dups_size":0,"ondisk_log_size":4961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40630864,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4960,"num_write_kb":39679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5092","reported_seq":5166,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.143274+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:29.143274+0000","last_peered":"2024-09-11T04:58:29.143274+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:29.143274+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41704016,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":40727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5084","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.647675+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:32.647675+0000","last_peered":"2024-09-11T04:58:32.647675+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:32.647675+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41612448,"num_objects":5080,"num_object_clones":0,"num_object_copies":20320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5082,"num_write_kb":40652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5020","reported_seq":5097,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:29.269507+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:29.269507+0000","last_peered":"2024-09-11T04:58:29.269507+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:29.269507+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41096352,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":40141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5171","reported_seq":5234,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.636925+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:31.636925+0000","last_peered":"2024-09-11T04:58:31.636925+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:31.636925+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42316960,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":41340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":665559232,"num_objects":81250,"num_object_clones":0,"num_object_copies":325000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81277,"num_write_kb":650098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1186426880,"data_stored":1186426880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81304,"ondisk_log_size":81304,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570676,"kb_used_data":326564,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93801164,"statfs":{"total":96636764160,"available":96052391936,"internally_reserved":0,"allocated":334401536,"data_stored":334230726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574648,"kb_used_data":322728,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93797192,"statfs":{"total":96636764160,"available":96048324608,"internally_reserved":0,"allocated":330473472,"data_stored":330302662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,6,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378312,"kb_used_data":184248,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93993528,"statfs":{"total":96636764160,"available":96249372672,"internally_reserved":0,"allocated":188669952,"data_stored":188499142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198700754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577456,"kb_used_data":325088,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93794384,"statfs":{"total":96636764160,"available":96045449216,"internally_reserved":0,"allocated":332890112,"data_stored":332719302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332976128,"data_stored":332976128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188858368,"data_stored":188858368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330399744,"data_stored":330399744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334192640,"data_stored":334192640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:35.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:35.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 172 20 82474 82454 3.74461 2.14844 0.0710794 0.0417127 2024-09-11T04:58:36.282 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:36.283 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:36.313 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-11T04:58:35.671318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668614848,"num_objects":81623,"num_object_clones":0,"num_object_copies":326492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81650,"num_write_kb":653082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81677,"ondisk_log_size":81677,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2110852,"kb_used_data":1164228,"kb_used_omap":14,"kb_used_meta":946545,"kb_avail":375376508,"statfs":{"total":386547056640,"available":384385544192,"internally_reserved":0,"allocated":1192169472,"data_stored":1191486232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":969262646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,6,13,17,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26492928,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":25872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006771"},"pg_stats":[{"pgid":"1.f","version":"31'5039","reported_seq":5081,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.581185+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:32.581185+0000","last_peered":"2024-09-11T04:58:32.581185+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:32.581185+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41279488,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5039,"num_write_kb":40312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5105","reported_seq":5167,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.564057+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:31.564057+0000","last_peered":"2024-09-11T04:58:31.564057+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:31.564057+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41784480,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5103,"num_write_kb":40820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4952","reported_seq":5027,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:34.272690+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:34.272690+0000","last_peered":"2024-09-11T04:58:34.272690+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:34.272690+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40540752,"num_objects":4949,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":39598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2729}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5068","reported_seq":5128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.528335+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:31.528335+0000","last_peered":"2024-09-11T04:58:31.528335+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:31.528335+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41481376,"num_objects":5064,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":40517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5074","reported_seq":5128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.567115+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:32.567115+0000","last_peered":"2024-09-11T04:58:32.567115+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:32.567115+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41538720,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5072,"num_write_kb":40573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5146","reported_seq":5187,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.640057+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:32.640057+0000","last_peered":"2024-09-11T04:58:32.640057+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:32.640057+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5146,"log_dups_size":0,"ondisk_log_size":5146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42156032,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5146,"num_write_kb":41168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5129","reported_seq":5192,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.557383+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:31.557383+0000","last_peered":"2024-09-11T04:58:31.557383+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:31.557383+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41964704,"num_objects":5123,"num_object_clones":0,"num_object_copies":20492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5127,"num_write_kb":40996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5176","reported_seq":5238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.595343+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:31.595343+0000","last_peered":"2024-09-11T04:58:31.595343+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:31.595343+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5176,"log_dups_size":0,"ondisk_log_size":5176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42357920,"num_objects":5171,"num_object_clones":0,"num_object_copies":20684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5174,"num_write_kb":41380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2810}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5182","reported_seq":5237,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.667131+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:32.667131+0000","last_peered":"2024-09-11T04:58:32.667131+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:32.667131+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5182,"log_dups_size":0,"ondisk_log_size":5182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42379584,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5179,"num_write_kb":41409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5159","reported_seq":5236,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:34.307030+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:34.307030+0000","last_peered":"2024-09-11T04:58:34.307030+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:34.307030+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42243232,"num_objects":5157,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":41254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5139","reported_seq":5195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.365453+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:32.365453+0000","last_peered":"2024-09-11T04:58:32.365453+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:32.365453+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42053360,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4961","reported_seq":7818,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.679714+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:32.679714+0000","last_peered":"2024-09-11T04:58:32.679714+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:32.679714+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4961,"log_dups_size":0,"ondisk_log_size":4961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40630864,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4960,"num_write_kb":39679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5183","reported_seq":5256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:34.274813+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:34.274813+0000","last_peered":"2024-09-11T04:58:34.274813+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:34.274813+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42449488,"num_objects":5182,"num_object_clones":0,"num_object_copies":20728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5182,"num_write_kb":41455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2850}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5084","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.647675+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:32.647675+0000","last_peered":"2024-09-11T04:58:32.647675+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:32.647675+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41612448,"num_objects":5080,"num_object_clones":0,"num_object_copies":20320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5082,"num_write_kb":40652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5109","reported_seq":5187,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:34.232289+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:34.232289+0000","last_peered":"2024-09-11T04:58:34.232289+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:34.232289+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41825440,"num_objects":5106,"num_object_clones":0,"num_object_copies":20424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":40853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2788}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5171","reported_seq":5234,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:31.636925+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:31.636925+0000","last_peered":"2024-09-11T04:58:31.636925+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:31.636925+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42316960,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":41340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":668614848,"num_objects":81623,"num_object_clones":0,"num_object_copies":326492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81650,"num_write_kb":653082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1192251392,"data_stored":1192251392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81677,"ondisk_log_size":81677,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570676,"kb_used_data":326564,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93801164,"statfs":{"total":96636764160,"available":96052391936,"internally_reserved":0,"allocated":334401536,"data_stored":334230726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584408,"kb_used_data":328328,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93787432,"statfs":{"total":96636764160,"available":96038330368,"internally_reserved":0,"allocated":336207872,"data_stored":336037062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378312,"kb_used_data":184248,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93993528,"statfs":{"total":96636764160,"available":96249372672,"internally_reserved":0,"allocated":188669952,"data_stored":188499142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198700754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577456,"kb_used_data":325088,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93794384,"statfs":{"total":96636764160,"available":96045449216,"internally_reserved":0,"allocated":332890112,"data_stored":332719302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332976128,"data_stored":332976128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188858368,"data_stored":188858368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336224256,"data_stored":336224256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334192640,"data_stored":334192640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:36.314 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5039", 'reported_seq': 5081, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:32.581185+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:32.581185+0000', 'last_peered': '2024-09-11T04:58:32.581185+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:32.581185+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5039, 'log_dups_size': 0, 'ondisk_log_size': 5039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41279488, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 20156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5039, 'num_write_kb': 40312, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2727}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5105", 'reported_seq': 5167, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:31.564057+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:31.564057+0000', 'last_peered': '2024-09-11T04:58:31.564057+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:31.564057+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5105, 'log_dups_size': 0, 'ondisk_log_size': 5105, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41784480, 'num_objects': 5101, 'num_object_clones': 0, 'num_object_copies': 20404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5103, 'num_write_kb': 40820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4952", 'reported_seq': 5027, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:34.272690+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:34.272690+0000', 'last_peered': '2024-09-11T04:58:34.272690+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:34.272690+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4952, 'log_dups_size': 0, 'ondisk_log_size': 4952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40540752, 'num_objects': 4949, 'num_object_clones': 0, 'num_object_copies': 19796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4951, 'num_write_kb': 39598, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2729}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5068", 'reported_seq': 5128, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:31.528335+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:31.528335+0000', 'last_peered': '2024-09-11T04:58:31.528335+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:31.528335+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5068, 'log_dups_size': 0, 'ondisk_log_size': 5068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41481376, 'num_objects': 5064, 'num_object_clones': 0, 'num_object_copies': 20256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5066, 'num_write_kb': 40517, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:36.315 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5074", 'reported_seq': 5128, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:32.567115+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:32.567115+0000', 'last_peered': '2024-09-11T04:58:32.567115+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:32.567115+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5074, 'log_dups_size': 0, 'ondisk_log_size': 5074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41538720, 'num_objects': 5071, 'num_object_clones': 0, 'num_object_copies': 20284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5072, 'num_write_kb': 40573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2821}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5146", 'reported_seq': 5187, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:32.640057+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:32.640057+0000', 'last_peered': '2024-09-11T04:58:32.640057+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:32.640057+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5146, 'log_dups_size': 0, 'ondisk_log_size': 5146, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42156032, 'num_objects': 5146, 'num_object_clones': 0, 'num_object_copies': 20584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5146, 'num_write_kb': 41168, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2825}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5129", 'reported_seq': 5192, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:31.557383+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:31.557383+0000', 'last_peered': '2024-09-11T04:58:31.557383+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:31.557383+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5129, 'log_dups_size': 0, 'ondisk_log_size': 5129, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41964704, 'num_objects': 5123, 'num_object_clones': 0, 'num_object_copies': 20492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5127, 'num_write_kb': 40996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2806}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5176", 'reported_seq': 5238, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:31.595343+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:31.595343+0000', 'last_peered': '2024-09-11T04:58:31.595343+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:31.595343+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5176, 'log_dups_size': 0, 'ondisk_log_size': 5176, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42357920, 'num_objects': 5171, 'num_object_clones': 0, 'num_object_copies': 20684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5174, 'num_write_kb': 41380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2810}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5182", 'reported_seq': 5237, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:32.667131+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:32.667131+0000', 'last_peered': '2024-09-11T04:58:32.667131+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:32.667131+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5182, 'log_dups_size': 0, 'ondisk_log_size': 5182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42379584, 'num_objects': 5174, 'num_object_clones': 0, 'num_object_copies': 20696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5179, 'num_write_kb': 41409, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2780}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:36.316 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5159", 'reported_seq': 5236, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:34.307030+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:34.307030+0000', 'last_peered': '2024-09-11T04:58:34.307030+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:34.307030+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5159, 'log_dups_size': 0, 'ondisk_log_size': 5159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42243232, 'num_objects': 5157, 'num_object_clones': 0, 'num_object_copies': 20628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5157, 'num_write_kb': 41254, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2839}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5139", 'reported_seq': 5195, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:32.365453+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:32.365453+0000', 'last_peered': '2024-09-11T04:58:32.365453+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:32.365453+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5139, 'log_dups_size': 0, 'ondisk_log_size': 5139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42053360, 'num_objects': 5134, 'num_object_clones': 0, 'num_object_copies': 20536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5136, 'num_write_kb': 41076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4961", 'reported_seq': 7818, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:32.679714+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:32.679714+0000', 'last_peered': '2024-09-11T04:58:32.679714+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:32.679714+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 4961, 'log_dups_size': 0, 'ondisk_log_size': 4961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40630864, 'num_objects': 4960, 'num_object_clones': 0, 'num_object_copies': 19840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 826, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4960, 'num_write_kb': 39679, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1411, 'num_bytes_recovered': 11542528, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5183", 'reported_seq': 5256, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:34.274813+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:34.274813+0000', 'last_peered': '2024-09-11T04:58:34.274813+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:34.274813+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5183, 'log_dups_size': 0, 'ondisk_log_size': 5183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42449488, 'num_objects': 5182, 'num_object_clones': 0, 'num_object_copies': 20728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5182, 'num_write_kb': 41455, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2850}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5084", 'reported_seq': 5134, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:32.647675+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:32.647675+0000', 'last_peered': '2024-09-11T04:58:32.647675+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:32.647675+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5084, 'log_dups_size': 0, 'ondisk_log_size': 5084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41612448, 'num_objects': 5080, 'num_object_clones': 0, 'num_object_copies': 20320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5082, 'num_write_kb': 40652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2793}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:36.317 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5109", 'reported_seq': 5187, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:34.232289+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:34.232289+0000', 'last_peered': '2024-09-11T04:58:34.232289+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:34.232289+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5109, 'log_dups_size': 0, 'ondisk_log_size': 5109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41825440, 'num_objects': 5106, 'num_object_clones': 0, 'num_object_copies': 20424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5107, 'num_write_kb': 40853, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2788}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:36.318 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:36.318 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5171", 'reported_seq': 5234, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:31.636925+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:31.636925+0000', 'last_peered': '2024-09-11T04:58:31.636925+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:31.636925+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5171, 'log_dups_size': 0, 'ondisk_log_size': 5171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42316960, 'num_objects': 5166, 'num_object_clones': 0, 'num_object_copies': 20664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5169, 'num_write_kb': 41340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:36.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:36.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 173 20 82737 82717 3.73484 2.05469 0.0317773 0.0418111 2024-09-11T04:58:37.108 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:37.111 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35473/326492 objects degraded (10.865%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":244,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81623,"data_bytes":668614848,"bytes_used":2161512448,"bytes_avail":384385544192,"bytes_total":386547056640,"degraded_objects":35473,"degraded_total":326492,"degraded_ratio":0.10864891023363513,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":82556,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":2206498,"read_op_per_sec":0,"write_op_per_sec":269},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:37.133 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:37.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:37.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 174 20 83001 82981 3.72523 2.0625 0.0747587 0.0419276 2024-09-11T04:58:37.716 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:37.716 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:37.747 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-11T04:58:37.672305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":672686272,"num_objects":82120,"num_object_clones":0,"num_object_copies":328480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82147,"num_write_kb":657058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82174,"ondisk_log_size":82174,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2130844,"kb_used_data":1175900,"kb_used_omap":14,"kb_used_meta":954865,"kb_avail":375356516,"statfs":{"total":386547056640,"available":384365072384,"internally_reserved":0,"allocated":1204121600,"data_stored":1203438360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":977782326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,8,6,19,21,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27869184,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3402,"num_write_kb":27216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006862"},"pg_stats":[{"pgid":"1.f","version":"31'5039","reported_seq":5081,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.581185+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:32.581185+0000","last_peered":"2024-09-11T04:58:32.581185+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:32.581185+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41279488,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5039,"num_write_kb":40312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5223","reported_seq":5285,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.671824+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:36.671824+0000","last_peered":"2024-09-11T04:58:36.671824+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:36.671824+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42751136,"num_objects":5219,"num_object_clones":0,"num_object_copies":20876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4952","reported_seq":5027,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:34.272690+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:34.272690+0000","last_peered":"2024-09-11T04:58:34.272690+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:34.272690+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40540752,"num_objects":4949,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":39598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2729}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5150","reported_seq":5213,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.401662+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:36.401662+0000","last_peered":"2024-09-11T04:58:36.401662+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:36.401662+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5150,"log_dups_size":0,"ondisk_log_size":5150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42153120,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":41173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5074","reported_seq":5128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.567115+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:32.567115+0000","last_peered":"2024-09-11T04:58:32.567115+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:32.567115+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41538720,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5072,"num_write_kb":40573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5146","reported_seq":5187,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.640057+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:32.640057+0000","last_peered":"2024-09-11T04:58:32.640057+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:32.640057+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5146,"log_dups_size":0,"ondisk_log_size":5146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42156032,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5146,"num_write_kb":41168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5226","reported_seq":5289,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.519321+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:36.519321+0000","last_peered":"2024-09-11T04:58:36.519321+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:36.519321+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2903}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5277","reported_seq":5339,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.446091+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:36.446091+0000","last_peered":"2024-09-11T04:58:36.446091+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:36.446091+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":42188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5182","reported_seq":5237,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.667131+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:32.667131+0000","last_peered":"2024-09-11T04:58:32.667131+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:32.667131+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5182,"log_dups_size":0,"ondisk_log_size":5182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42379584,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5179,"num_write_kb":41409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5159","reported_seq":5236,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:34.307030+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:34.307030+0000","last_peered":"2024-09-11T04:58:34.307030+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:34.307030+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42243232,"num_objects":5157,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":41254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5139","reported_seq":5195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.365453+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:32.365453+0000","last_peered":"2024-09-11T04:58:32.365453+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:32.365453+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42053360,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4961","reported_seq":7818,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.679714+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:32.679714+0000","last_peered":"2024-09-11T04:58:32.679714+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:32.679714+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":4961,"log_dups_size":0,"ondisk_log_size":4961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40630864,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4960,"num_write_kb":39679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5183","reported_seq":5256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:34.274813+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:34.274813+0000","last_peered":"2024-09-11T04:58:34.274813+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:34.274813+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42449488,"num_objects":5182,"num_object_clones":0,"num_object_copies":20728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5182,"num_write_kb":41455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2850}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5084","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:32.647675+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:32.647675+0000","last_peered":"2024-09-11T04:58:32.647675+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:32.647675+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41612448,"num_objects":5080,"num_object_clones":0,"num_object_copies":20320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5082,"num_write_kb":40652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5109","reported_seq":5187,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:34.232289+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:34.232289+0000","last_peered":"2024-09-11T04:58:34.232289+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:34.232289+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41825440,"num_objects":5106,"num_object_clones":0,"num_object_copies":20424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":40853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2788}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5270","reported_seq":5333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.689151+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:36.689151+0000","last_peered":"2024-09-11T04:58:36.689151+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:36.689151+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43127968,"num_objects":5265,"num_object_clones":0,"num_object_copies":21060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":42132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2954}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":672686272,"num_objects":82120,"num_object_clones":0,"num_object_copies":328480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82147,"num_write_kb":657058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204273152,"data_stored":1204273152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82174,"ondisk_log_size":82174,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570676,"kb_used_data":326564,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93801164,"statfs":{"total":96636764160,"available":96052391936,"internally_reserved":0,"allocated":334401536,"data_stored":334230726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584408,"kb_used_data":328328,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93787432,"statfs":{"total":96636764160,"available":96038330368,"internally_reserved":0,"allocated":336207872,"data_stored":336037062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388444,"kb_used_data":190220,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93983396,"statfs":{"total":96636764160,"available":96238997504,"internally_reserved":0,"allocated":194785280,"data_stored":194614470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":202960594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587316,"kb_used_data":330788,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93784524,"statfs":{"total":96636764160,"available":96035352576,"internally_reserved":0,"allocated":338726912,"data_stored":338556102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":338931712,"data_stored":338931712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194924544,"data_stored":194924544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336224256,"data_stored":336224256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334192640,"data_stored":334192640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:38.711 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 175 20 83306 83286 3.71756 2.38281 0.0397312 0.0420157 2024-09-11T04:58:39.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 176 20 83607 83587 3.70979 2.35156 0.0652571 0.0421067 2024-09-11T04:58:40.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 177 20 83911 83891 3.70225 2.375 0.0413197 0.0421907 2024-09-11T04:58:40.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:41.419 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:41.419 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:41.454 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-11T04:58:39.673226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":680378560,"num_objects":83059,"num_object_clones":0,"num_object_copies":332236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83086,"num_write_kb":664570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83113,"ondisk_log_size":83113,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2151804,"kb_used_data":1187132,"kb_used_omap":14,"kb_used_meta":964593,"kb_avail":375335556,"statfs":{"total":386547056640,"available":384343609344,"internally_reserved":0,"allocated":1215623168,"data_stored":1214939928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":987743798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,5,8,4,12,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26517504,"num_objects":3237,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":25896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006918"},"pg_stats":[{"pgid":"1.f","version":"31'5122","reported_seq":5163,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.629174+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:37.629174+0000","last_peered":"2024-09-11T04:58:37.629174+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:37.629174+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41959424,"num_objects":5122,"num_object_clones":0,"num_object_copies":20488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5223","reported_seq":5285,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.671824+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:36.671824+0000","last_peered":"2024-09-11T04:58:36.671824+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:36.671824+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42751136,"num_objects":5219,"num_object_clones":0,"num_object_copies":20876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5052","reported_seq":5127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.302936+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:39.302936+0000","last_peered":"2024-09-11T04:58:39.302936+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:39.302936+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41359952,"num_objects":5049,"num_object_clones":0,"num_object_copies":20196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":40398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5150","reported_seq":5213,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.401662+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:36.401662+0000","last_peered":"2024-09-11T04:58:36.401662+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:36.401662+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5150,"log_dups_size":0,"ondisk_log_size":5150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42153120,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":41173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5175","reported_seq":5229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.632693+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:37.632693+0000","last_peered":"2024-09-11T04:58:37.632693+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:37.632693+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42366112,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":41381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5214","reported_seq":5255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.531477+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:37.531477+0000","last_peered":"2024-09-11T04:58:37.531477+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:37.531477+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42713088,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5214,"num_write_kb":41712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5226","reported_seq":5289,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.519321+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:36.519321+0000","last_peered":"2024-09-11T04:58:36.519321+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:36.519321+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2903}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5277","reported_seq":5339,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.446091+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:36.446091+0000","last_peered":"2024-09-11T04:58:36.446091+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:36.446091+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":42188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5262","reported_seq":5317,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.593106+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:37.593106+0000","last_peered":"2024-09-11T04:58:37.593106+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:37.593106+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43034944,"num_objects":5254,"num_object_clones":0,"num_object_copies":21016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5237","reported_seq":5314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.304814+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:39.304814+0000","last_peered":"2024-09-11T04:58:39.304814+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:39.304814+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42882208,"num_objects":5235,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":41878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2917}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5227","reported_seq":5283,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.698667+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:37.698667+0000","last_peered":"2024-09-11T04:58:37.698667+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:37.698667+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42774256,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5061","reported_seq":8033,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.726374+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:37.726374+0000","last_peered":"2024-09-11T04:58:37.726374+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:37.726374+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41450064,"num_objects":5060,"num_object_clones":0,"num_object_copies":20240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":40479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1469,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4292}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5265","reported_seq":5337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.256730+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:39.256730+0000","last_peered":"2024-09-11T04:58:39.256730+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:39.256730+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43121232,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5158","reported_seq":5207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.644946+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:37.644946+0000","last_peered":"2024-09-11T04:58:37.644946+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:37.644946+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5158,"log_dups_size":0,"ondisk_log_size":5158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42218656,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5156,"num_write_kb":41244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2867}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5194","reported_seq":5272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.229289+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:39.229289+0000","last_peered":"2024-09-11T04:58:39.229289+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:39.229289+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42521760,"num_objects":5191,"num_object_clones":0,"num_object_copies":20764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":41533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5270","reported_seq":5333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.689151+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:36.689151+0000","last_peered":"2024-09-11T04:58:36.689151+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:36.689151+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43127968,"num_objects":5265,"num_object_clones":0,"num_object_copies":21060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":42132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2954}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":680378560,"num_objects":83059,"num_object_clones":0,"num_object_copies":332236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83086,"num_write_kb":664570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215827968,"data_stored":1215827968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83113,"ondisk_log_size":83113,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580504,"kb_used_data":332232,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93791336,"statfs":{"total":96636764160,"available":96042328064,"internally_reserved":0,"allocated":340205568,"data_stored":340034758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595540,"kb_used_data":333892,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93776300,"statfs":{"total":96636764160,"available":96026931200,"internally_reserved":0,"allocated":341905408,"data_stored":341734598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388444,"kb_used_data":190220,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93983396,"statfs":{"total":96636764160,"available":96238997504,"internally_reserved":0,"allocated":194785280,"data_stored":194614470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":202960594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587316,"kb_used_data":330788,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93784524,"statfs":{"total":96636764160,"available":96035352576,"internally_reserved":0,"allocated":338726912,"data_stored":338556102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":338931712,"data_stored":338931712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194924544,"data_stored":194924544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341975040,"data_stored":341975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339996672,"data_stored":339996672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:41.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:41.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 178 20 84184 84164 3.69343 2.13281 0.0577001 0.0422944 2024-09-11T04:58:42.104 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:42.122 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-11T04:58:41.673962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":680378560,"num_objects":83059,"num_object_clones":0,"num_object_copies":332236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83086,"num_write_kb":664570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83113,"ondisk_log_size":83113,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2151804,"kb_used_data":1187132,"kb_used_omap":14,"kb_used_meta":964593,"kb_avail":375335556,"statfs":{"total":386547056640,"available":384343609344,"internally_reserved":0,"allocated":1215623168,"data_stored":1214939928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":987743798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,5,8,4,12,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23429120,"num_objects":2860,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":22880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006683"},"pg_stats":[{"pgid":"1.f","version":"31'5122","reported_seq":5163,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.629174+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:37.629174+0000","last_peered":"2024-09-11T04:58:37.629174+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:37.629174+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41959424,"num_objects":5122,"num_object_clones":0,"num_object_copies":20488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5223","reported_seq":5285,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.671824+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:36.671824+0000","last_peered":"2024-09-11T04:58:36.671824+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:36.671824+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42751136,"num_objects":5219,"num_object_clones":0,"num_object_copies":20876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5052","reported_seq":5127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.302936+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:39.302936+0000","last_peered":"2024-09-11T04:58:39.302936+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:39.302936+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41359952,"num_objects":5049,"num_object_clones":0,"num_object_copies":20196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":40398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5150","reported_seq":5213,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.401662+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:36.401662+0000","last_peered":"2024-09-11T04:58:36.401662+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:36.401662+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5150,"log_dups_size":0,"ondisk_log_size":5150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42153120,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":41173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5175","reported_seq":5229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.632693+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:37.632693+0000","last_peered":"2024-09-11T04:58:37.632693+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:37.632693+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42366112,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":41381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5214","reported_seq":5255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.531477+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:37.531477+0000","last_peered":"2024-09-11T04:58:37.531477+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:37.531477+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42713088,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5214,"num_write_kb":41712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5226","reported_seq":5289,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.519321+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:36.519321+0000","last_peered":"2024-09-11T04:58:36.519321+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:36.519321+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2903}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5277","reported_seq":5339,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.446091+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:36.446091+0000","last_peered":"2024-09-11T04:58:36.446091+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:36.446091+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":42188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5262","reported_seq":5317,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.593106+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:37.593106+0000","last_peered":"2024-09-11T04:58:37.593106+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:37.593106+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43034944,"num_objects":5254,"num_object_clones":0,"num_object_copies":21016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5237","reported_seq":5314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.304814+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:39.304814+0000","last_peered":"2024-09-11T04:58:39.304814+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:39.304814+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42882208,"num_objects":5235,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":41878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2917}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5227","reported_seq":5283,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.698667+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:37.698667+0000","last_peered":"2024-09-11T04:58:37.698667+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:37.698667+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42774256,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5061","reported_seq":8033,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.726374+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:37.726374+0000","last_peered":"2024-09-11T04:58:37.726374+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:37.726374+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41450064,"num_objects":5060,"num_object_clones":0,"num_object_copies":20240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":40479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1469,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4292}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5265","reported_seq":5337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.256730+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:39.256730+0000","last_peered":"2024-09-11T04:58:39.256730+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:39.256730+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43121232,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5158","reported_seq":5207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.644946+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:37.644946+0000","last_peered":"2024-09-11T04:58:37.644946+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:37.644946+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5158,"log_dups_size":0,"ondisk_log_size":5158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42218656,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5156,"num_write_kb":41244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2867}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5194","reported_seq":5272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.229289+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:39.229289+0000","last_peered":"2024-09-11T04:58:39.229289+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:39.229289+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42521760,"num_objects":5191,"num_object_clones":0,"num_object_copies":20764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":41533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5270","reported_seq":5333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.689151+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:36.689151+0000","last_peered":"2024-09-11T04:58:36.689151+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:36.689151+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43127968,"num_objects":5265,"num_object_clones":0,"num_object_copies":21060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":42132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2954}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":680378560,"num_objects":83059,"num_object_clones":0,"num_object_copies":332236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83086,"num_write_kb":664570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215827968,"data_stored":1215827968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83113,"ondisk_log_size":83113,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580504,"kb_used_data":332232,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93791336,"statfs":{"total":96636764160,"available":96042328064,"internally_reserved":0,"allocated":340205568,"data_stored":340034758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595540,"kb_used_data":333892,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93776300,"statfs":{"total":96636764160,"available":96026931200,"internally_reserved":0,"allocated":341905408,"data_stored":341734598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388444,"kb_used_data":190220,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93983396,"statfs":{"total":96636764160,"available":96238997504,"internally_reserved":0,"allocated":194785280,"data_stored":194614470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":202960594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587316,"kb_used_data":330788,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93784524,"statfs":{"total":96636764160,"available":96035352576,"internally_reserved":0,"allocated":338726912,"data_stored":338556102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":338931712,"data_stored":338931712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194924544,"data_stored":194924544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341975040,"data_stored":341975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339996672,"data_stored":339996672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:42.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:42.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5122", 'reported_seq': 5163, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:37.629174+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:37.629174+0000', 'last_peered': '2024-09-11T04:58:37.629174+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:37.629174+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5122, 'log_dups_size': 0, 'ondisk_log_size': 5122, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41959424, 'num_objects': 5122, 'num_object_clones': 0, 'num_object_copies': 20488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5122, 'num_write_kb': 40976, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2810}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5223", 'reported_seq': 5285, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:36.671824+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:36.671824+0000', 'last_peered': '2024-09-11T04:58:36.671824+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:36.671824+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5223, 'log_dups_size': 0, 'ondisk_log_size': 5223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42751136, 'num_objects': 5219, 'num_object_clones': 0, 'num_object_copies': 20876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5221, 'num_write_kb': 41764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5052", 'reported_seq': 5127, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:39.302936+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:39.302936+0000', 'last_peered': '2024-09-11T04:58:39.302936+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:39.302936+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5052, 'log_dups_size': 0, 'ondisk_log_size': 5052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41359952, 'num_objects': 5049, 'num_object_clones': 0, 'num_object_copies': 20196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5051, 'num_write_kb': 40398, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2829}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5150", 'reported_seq': 5213, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:36.401662+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:36.401662+0000', 'last_peered': '2024-09-11T04:58:36.401662+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:36.401662+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5150, 'log_dups_size': 0, 'ondisk_log_size': 5150, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42153120, 'num_objects': 5146, 'num_object_clones': 0, 'num_object_copies': 20584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5148, 'num_write_kb': 41173, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2868}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5175", 'reported_seq': 5229, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:37.632693+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:37.632693+0000', 'last_peered': '2024-09-11T04:58:37.632693+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:37.632693+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5175, 'log_dups_size': 0, 'ondisk_log_size': 5175, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42366112, 'num_objects': 5172, 'num_object_clones': 0, 'num_object_copies': 20688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5173, 'num_write_kb': 41381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2922}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:42.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:42.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5214", 'reported_seq': 5255, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:37.531477+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:37.531477+0000', 'last_peered': '2024-09-11T04:58:37.531477+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:37.531477+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5214, 'log_dups_size': 0, 'ondisk_log_size': 5214, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42713088, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5214, 'num_write_kb': 41712, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2893}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:42.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:42.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5226", 'reported_seq': 5289, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:36.519321+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:36.519321+0000', 'last_peered': '2024-09-11T04:58:36.519321+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:36.519321+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5226, 'log_dups_size': 0, 'ondisk_log_size': 5226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42759328, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 20880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5224, 'num_write_kb': 41772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2903}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:42.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:42.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5277", 'reported_seq': 5339, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:36.446091+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:36.446091+0000', 'last_peered': '2024-09-11T04:58:36.446091+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:36.446091+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5277, 'log_dups_size': 0, 'ondisk_log_size': 5277, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43185312, 'num_objects': 5272, 'num_object_clones': 0, 'num_object_copies': 21088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5275, 'num_write_kb': 42188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:42.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:42.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5262", 'reported_seq': 5317, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:37.593106+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:37.593106+0000', 'last_peered': '2024-09-11T04:58:37.593106+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:37.593106+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5262, 'log_dups_size': 0, 'ondisk_log_size': 5262, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43034944, 'num_objects': 5254, 'num_object_clones': 0, 'num_object_copies': 21016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5259, 'num_write_kb': 42049, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2860}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:42.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:42.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5237", 'reported_seq': 5314, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:39.304814+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:39.304814+0000', 'last_peered': '2024-09-11T04:58:39.304814+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:39.304814+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5237, 'log_dups_size': 0, 'ondisk_log_size': 5237, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42882208, 'num_objects': 5235, 'num_object_clones': 0, 'num_object_copies': 20940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5235, 'num_write_kb': 41878, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2917}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:42.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:42.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5227", 'reported_seq': 5283, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:37.698667+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:37.698667+0000', 'last_peered': '2024-09-11T04:58:37.698667+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:37.698667+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5227, 'log_dups_size': 0, 'ondisk_log_size': 5227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42774256, 'num_objects': 5222, 'num_object_clones': 0, 'num_object_copies': 20888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5224, 'num_write_kb': 41780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2899}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:42.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:42.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5061", 'reported_seq': 8033, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:37.726374+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:37.726374+0000', 'last_peered': '2024-09-11T04:58:37.726374+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:37.726374+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5061, 'log_dups_size': 0, 'ondisk_log_size': 5061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41450064, 'num_objects': 5060, 'num_object_clones': 0, 'num_object_copies': 20240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 768, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5060, 'num_write_kb': 40479, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1469, 'num_bytes_recovered': 12017664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4292}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:42.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:42.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5265", 'reported_seq': 5337, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:39.256730+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:39.256730+0000', 'last_peered': '2024-09-11T04:58:39.256730+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:39.256730+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5265, 'log_dups_size': 0, 'ondisk_log_size': 5265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43121232, 'num_objects': 5264, 'num_object_clones': 0, 'num_object_copies': 21056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5264, 'num_write_kb': 42111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2932}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:42.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:42.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5158", 'reported_seq': 5207, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:37.644946+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:37.644946+0000', 'last_peered': '2024-09-11T04:58:37.644946+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:37.644946+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5158, 'log_dups_size': 0, 'ondisk_log_size': 5158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42218656, 'num_objects': 5154, 'num_object_clones': 0, 'num_object_copies': 20616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5156, 'num_write_kb': 41244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2867}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:42.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:42.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5194", 'reported_seq': 5272, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:39.229289+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:39.229289+0000', 'last_peered': '2024-09-11T04:58:39.229289+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:39.229289+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5194, 'log_dups_size': 0, 'ondisk_log_size': 5194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42521760, 'num_objects': 5191, 'num_object_clones': 0, 'num_object_copies': 20764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5192, 'num_write_kb': 41533, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2873}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:42.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:42.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5270", 'reported_seq': 5333, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:36.689151+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:36.689151+0000', 'last_peered': '2024-09-11T04:58:36.689151+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:36.689151+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5270, 'log_dups_size': 0, 'ondisk_log_size': 5270, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43127968, 'num_objects': 5265, 'num_object_clones': 0, 'num_object_copies': 21060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5268, 'num_write_kb': 42132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:42.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:42.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 179 20 84444 84424 3.68415 2.03125 0.0690159 0.0423956 2024-09-11T04:58:42.877 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:42.877 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35415/332236 objects degraded (10.660%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":249,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83059,"data_bytes":680378560,"bytes_used":2203447296,"bytes_avail":384343609344,"bytes_total":386547056640,"degraded_objects":35415,"degraded_total":332236,"degraded_ratio":0.10659591374805862,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":77098,"recovering_keys_per_sec":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"write_bytes_sec":1951339,"read_op_per_sec":0,"write_op_per_sec":238},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:42.902 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:42.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:43.497 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:43.522 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-11T04:58:41.673962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":680378560,"num_objects":83059,"num_object_clones":0,"num_object_copies":332236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83086,"num_write_kb":664570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83113,"ondisk_log_size":83113,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2151804,"kb_used_data":1187132,"kb_used_omap":14,"kb_used_meta":964593,"kb_avail":375335556,"statfs":{"total":386547056640,"available":384343609344,"internally_reserved":0,"allocated":1215623168,"data_stored":1214939928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":987743798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,5,8,4,12,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23429120,"num_objects":2860,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":22880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006683"},"pg_stats":[{"pgid":"1.f","version":"31'5122","reported_seq":5163,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.629174+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:37.629174+0000","last_peered":"2024-09-11T04:58:37.629174+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:37.629174+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41959424,"num_objects":5122,"num_object_clones":0,"num_object_copies":20488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5223","reported_seq":5285,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.671824+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:36.671824+0000","last_peered":"2024-09-11T04:58:36.671824+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:36.671824+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42751136,"num_objects":5219,"num_object_clones":0,"num_object_copies":20876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5052","reported_seq":5127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.302936+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:39.302936+0000","last_peered":"2024-09-11T04:58:39.302936+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:39.302936+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41359952,"num_objects":5049,"num_object_clones":0,"num_object_copies":20196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":40398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5150","reported_seq":5213,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.401662+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:36.401662+0000","last_peered":"2024-09-11T04:58:36.401662+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:36.401662+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5150,"log_dups_size":0,"ondisk_log_size":5150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42153120,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":41173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2868}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5175","reported_seq":5229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.632693+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:37.632693+0000","last_peered":"2024-09-11T04:58:37.632693+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:37.632693+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42366112,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":41381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5214","reported_seq":5255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.531477+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:37.531477+0000","last_peered":"2024-09-11T04:58:37.531477+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:37.531477+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42713088,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5214,"num_write_kb":41712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5226","reported_seq":5289,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.519321+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:36.519321+0000","last_peered":"2024-09-11T04:58:36.519321+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:36.519321+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759328,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2903}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5277","reported_seq":5339,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.446091+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:36.446091+0000","last_peered":"2024-09-11T04:58:36.446091+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:36.446091+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":42188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5262","reported_seq":5317,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.593106+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:37.593106+0000","last_peered":"2024-09-11T04:58:37.593106+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:37.593106+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43034944,"num_objects":5254,"num_object_clones":0,"num_object_copies":21016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5237","reported_seq":5314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.304814+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:39.304814+0000","last_peered":"2024-09-11T04:58:39.304814+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:39.304814+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42882208,"num_objects":5235,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":41878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2917}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5227","reported_seq":5283,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.698667+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:37.698667+0000","last_peered":"2024-09-11T04:58:37.698667+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:37.698667+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42774256,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5061","reported_seq":8033,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.726374+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:37.726374+0000","last_peered":"2024-09-11T04:58:37.726374+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:37.726374+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41450064,"num_objects":5060,"num_object_clones":0,"num_object_copies":20240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":40479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1469,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4292}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5265","reported_seq":5337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.256730+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:39.256730+0000","last_peered":"2024-09-11T04:58:39.256730+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:39.256730+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43121232,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5158","reported_seq":5207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:37.644946+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:37.644946+0000","last_peered":"2024-09-11T04:58:37.644946+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:37.644946+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5158,"log_dups_size":0,"ondisk_log_size":5158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42218656,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5156,"num_write_kb":41244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2867}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5194","reported_seq":5272,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:39.229289+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:39.229289+0000","last_peered":"2024-09-11T04:58:39.229289+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:39.229289+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42521760,"num_objects":5191,"num_object_clones":0,"num_object_copies":20764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":41533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5270","reported_seq":5333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:36.689151+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:36.689151+0000","last_peered":"2024-09-11T04:58:36.689151+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:36.689151+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43127968,"num_objects":5265,"num_object_clones":0,"num_object_copies":21060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":42132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2954}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":680378560,"num_objects":83059,"num_object_clones":0,"num_object_copies":332236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83086,"num_write_kb":664570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":12017664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215827968,"data_stored":1215827968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83113,"ondisk_log_size":83113,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580504,"kb_used_data":332232,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93791336,"statfs":{"total":96636764160,"available":96042328064,"internally_reserved":0,"allocated":340205568,"data_stored":340034758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595540,"kb_used_data":333892,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93776300,"statfs":{"total":96636764160,"available":96026931200,"internally_reserved":0,"allocated":341905408,"data_stored":341734598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388444,"kb_used_data":190220,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93983396,"statfs":{"total":96636764160,"available":96238997504,"internally_reserved":0,"allocated":194785280,"data_stored":194614470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":202960594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587316,"kb_used_data":330788,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93784524,"statfs":{"total":96636764160,"available":96035352576,"internally_reserved":0,"allocated":338726912,"data_stored":338556102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262663887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":338931712,"data_stored":338931712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194924544,"data_stored":194924544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341975040,"data_stored":341975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339996672,"data_stored":339996672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:43.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:58:43.710441+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0424995 2024-09-11T04:58:43.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:58:43.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 180 20 84714 84694 3.6754 2.10938 0.0314482 0.0424995 2024-09-11T04:58:44.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 181 20 85011 84991 3.66791 2.32031 0.0652157 0.0425855 2024-09-11T04:58:45.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 182 20 85312 85292 3.66068 2.35156 0.0461616 0.0426697 2024-09-11T04:58:46.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:46.710 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 183 20 85600 85580 3.65296 2.25 0.104154 0.0427588 2024-09-11T04:58:47.198 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:47.228 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-11T04:58:45.675792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692216000,"num_objects":84504,"num_object_clones":0,"num_object_copies":338016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84531,"num_write_kb":676130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84558,"ondisk_log_size":84558,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2229284,"kb_used_data":1210468,"kb_used_omap":14,"kb_used_meta":1018737,"kb_avail":375258076,"statfs":{"total":386547056640,"available":384264269824,"internally_reserved":0,"allocated":1239519232,"data_stored":1238835992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1043187254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,14,20,15,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26656768,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3254,"num_write_kb":26032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006749"},"pg_stats":[{"pgid":"1.f","version":"31'5215","reported_seq":5255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.727394+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:42.727394+0000","last_peered":"2024-09-11T04:58:42.727394+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:42.727394+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42721280,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":41720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5331","reported_seq":5393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:41.591807+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:41.591807+0000","last_peered":"2024-09-11T04:58:41.591807+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:41.591807+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43635872,"num_objects":5327,"num_object_clones":0,"num_object_copies":21308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":42628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5145","reported_seq":5220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.332938+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:44.332938+0000","last_peered":"2024-09-11T04:58:44.332938+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:44.332938+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42121808,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5144,"num_write_kb":41142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2922}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5245","reported_seq":5308,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:41.698888+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:41.698888+0000","last_peered":"2024-09-11T04:58:41.698888+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:41.698888+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42931360,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5243,"num_write_kb":41933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5261","reported_seq":5315,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.755309+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:42.755309+0000","last_peered":"2024-09-11T04:58:42.755309+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:42.755309+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43070624,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5306","reported_seq":5347,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.790024+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:42.790024+0000","last_peered":"2024-09-11T04:58:42.790024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:42.790024+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5306,"num_write_kb":42448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5316","reported_seq":5379,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:41.585313+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:41.585313+0000","last_peered":"2024-09-11T04:58:41.585313+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:41.585313+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43496608,"num_objects":5310,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5314,"num_write_kb":42492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2993}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5362","reported_seq":5424,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:41.779809+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:41.779809+0000","last_peered":"2024-09-11T04:58:41.779809+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:41.779809+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43881632,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5360,"num_write_kb":42868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2996}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5348","reported_seq":5404,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.497267+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:42.497267+0000","last_peered":"2024-09-11T04:58:42.497267+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:42.497267+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5348,"log_dups_size":0,"ondisk_log_size":5348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43739456,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5345,"num_write_kb":42737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5318","reported_seq":5395,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.283189+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:44.283189+0000","last_peered":"2024-09-11T04:58:44.283189+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:44.283189+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43545760,"num_objects":5316,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5316,"num_write_kb":42526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2998}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5316","reported_seq":5372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.698858+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:42.698858+0000","last_peered":"2024-09-11T04:58:42.698858+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:42.698858+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43503344,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5313,"num_write_kb":42492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5162","reported_seq":8249,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.767696+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:42.767696+0000","last_peered":"2024-09-11T04:58:42.767696+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:42.767696+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42277456,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":41287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1526,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5340","reported_seq":5414,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.167077+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:44.167077+0000","last_peered":"2024-09-11T04:58:44.167077+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:44.167077+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43735632,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":42711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3007}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5244","reported_seq":5294,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.591687+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:42.591687+0000","last_peered":"2024-09-11T04:58:42.591687+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:42.591687+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42923168,"num_objects":5240,"num_object_clones":0,"num_object_copies":20960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5242,"num_write_kb":41932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2953}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5276","reported_seq":5354,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.317451+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:44.317451+0000","last_peered":"2024-09-11T04:58:44.317451+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:44.317451+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43193504,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5274,"num_write_kb":42189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2955}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5373","reported_seq":5436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:41.767176+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:41.767176+0000","last_peered":"2024-09-11T04:58:41.767176+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:41.767176+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5373,"log_dups_size":0,"ondisk_log_size":5373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43971744,"num_objects":5368,"num_object_clones":0,"num_object_copies":21472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5371,"num_write_kb":42956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":692216000,"num_objects":84504,"num_object_clones":0,"num_object_copies":338016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84531,"num_write_kb":676130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1239814144,"data_stored":1239814144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84558,"ondisk_log_size":84558,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608556,"kb_used_data":338076,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93763284,"statfs":{"total":96636764160,"available":96013602816,"internally_reserved":0,"allocated":346189824,"data_stored":346019014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":276950736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,11,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623512,"kb_used_data":339656,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93748328,"statfs":{"total":96636764160,"available":95998287872,"internally_reserved":0,"allocated":347807744,"data_stored":347636934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398584,"kb_used_data":196200,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93973256,"statfs":{"total":96636764160,"available":96228614144,"internally_reserved":0,"allocated":200908800,"data_stored":200737990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207220434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,9,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598632,"kb_used_data":336536,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93773208,"statfs":{"total":96636764160,"available":96023764992,"internally_reserved":0,"allocated":344612864,"data_stored":344442054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,12,2,2,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344891392,"data_stored":344891392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201121792,"data_stored":201121792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":347795456,"data_stored":347795456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346005504,"data_stored":346005504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:47.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:47.710 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 184 20 85866 85846 3.6444 2.07812 0.052035 0.042858 2024-09-11T04:58:47.862 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:47.863 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:47.890 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-11T04:58:47.676467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":695845056,"num_objects":84947,"num_object_clones":0,"num_object_copies":339788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84974,"num_write_kb":679674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85001,"ondisk_log_size":85001,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2257228,"kb_used_data":1216204,"kb_used_omap":14,"kb_used_meta":1040945,"kb_avail":375230132,"statfs":{"total":386547056640,"available":384235655168,"internally_reserved":0,"allocated":1245392896,"data_stored":1244709656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1065928246},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,21,16,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27230208,"num_objects":3324,"num_object_clones":0,"num_object_copies":13296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005149"},"pg_stats":[{"pgid":"1.f","version":"31'5215","reported_seq":5255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.727394+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:42.727394+0000","last_peered":"2024-09-11T04:58:42.727394+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:42.727394+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42721280,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":41720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5429","reported_seq":5491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.756393+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:46.756393+0000","last_peered":"2024-09-11T04:58:46.756393+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:46.756393+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5429,"log_dups_size":0,"ondisk_log_size":5429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44438688,"num_objects":5425,"num_object_clones":0,"num_object_copies":21700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5427,"num_write_kb":43412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5145","reported_seq":5220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.332938+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:44.332938+0000","last_peered":"2024-09-11T04:58:44.332938+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:44.332938+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42121808,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5144,"num_write_kb":41142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2922}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5335","reported_seq":5397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.697158+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:46.697158+0000","last_peered":"2024-09-11T04:58:46.697158+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:46.697158+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5333,"num_write_kb":42653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5261","reported_seq":5315,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.755309+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:42.755309+0000","last_peered":"2024-09-11T04:58:42.755309+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:42.755309+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43070624,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5306","reported_seq":5347,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.790024+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:42.790024+0000","last_peered":"2024-09-11T04:58:42.790024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:42.790024+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5306,"num_write_kb":42448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5401","reported_seq":5464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.712915+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:46.712915+0000","last_peered":"2024-09-11T04:58:46.712915+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:46.712915+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44192928,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3078}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5448","reported_seq":5510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.426170+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:46.426170+0000","last_peered":"2024-09-11T04:58:46.426170+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:46.426170+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586144,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3082}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5348","reported_seq":5404,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.497267+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:42.497267+0000","last_peered":"2024-09-11T04:58:42.497267+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:42.497267+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5348,"log_dups_size":0,"ondisk_log_size":5348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43739456,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5345,"num_write_kb":42737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5318","reported_seq":5395,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.283189+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:44.283189+0000","last_peered":"2024-09-11T04:58:44.283189+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:44.283189+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43545760,"num_objects":5316,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5316,"num_write_kb":42526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2998}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5316","reported_seq":5372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.698858+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:42.698858+0000","last_peered":"2024-09-11T04:58:42.698858+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:42.698858+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43503344,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5313,"num_write_kb":42492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5162","reported_seq":8249,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.767696+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:42.767696+0000","last_peered":"2024-09-11T04:58:42.767696+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:42.767696+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42277456,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":41287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1526,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5340","reported_seq":5414,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.167077+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:44.167077+0000","last_peered":"2024-09-11T04:58:44.167077+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:44.167077+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43735632,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":42711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3007}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5244","reported_seq":5294,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.591687+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:42.591687+0000","last_peered":"2024-09-11T04:58:42.591687+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:42.591687+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42923168,"num_objects":5240,"num_object_clones":0,"num_object_copies":20960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5242,"num_write_kb":41932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2953}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5276","reported_seq":5354,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.317451+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:44.317451+0000","last_peered":"2024-09-11T04:58:44.317451+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:44.317451+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43193504,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5274,"num_write_kb":42189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2955}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5457","reported_seq":5520,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.698785+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:46.698785+0000","last_peered":"2024-09-11T04:58:46.698785+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:46.698785+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44659872,"num_objects":5452,"num_object_clones":0,"num_object_copies":21808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5455,"num_write_kb":43628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":695845056,"num_objects":84947,"num_object_clones":0,"num_object_copies":339788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84974,"num_write_kb":679674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1245638656,"data_stored":1245638656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85001,"ondisk_log_size":85001,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608556,"kb_used_data":338076,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93763284,"statfs":{"total":96636764160,"available":96013602816,"internally_reserved":0,"allocated":346189824,"data_stored":346019014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":276950736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,11,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623512,"kb_used_data":339656,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93748328,"statfs":{"total":96636764160,"available":95998287872,"internally_reserved":0,"allocated":347807744,"data_stored":347636934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398584,"kb_used_data":196200,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93973256,"statfs":{"total":96636764160,"available":96228614144,"internally_reserved":0,"allocated":200908800,"data_stored":200737990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207220434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,9,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626576,"kb_used_data":342272,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93745264,"statfs":{"total":96636764160,"available":95995150336,"internally_reserved":0,"allocated":350486528,"data_stored":350315718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350715904,"data_stored":350715904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201121792,"data_stored":201121792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":347795456,"data_stored":347795456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346005504,"data_stored":346005504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:47.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:47.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5215", 'reported_seq': 5255, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:42.727394+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:42.727394+0000', 'last_peered': '2024-09-11T04:58:42.727394+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:42.727394+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5215, 'log_dups_size': 0, 'ondisk_log_size': 5215, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42721280, 'num_objects': 5215, 'num_object_clones': 0, 'num_object_copies': 20860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5215, 'num_write_kb': 41720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2903}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:47.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:47.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5429", 'reported_seq': 5491, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:46.756393+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:46.756393+0000', 'last_peered': '2024-09-11T04:58:46.756393+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:46.756393+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5429, 'log_dups_size': 0, 'ondisk_log_size': 5429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44438688, 'num_objects': 5425, 'num_object_clones': 0, 'num_object_copies': 21700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5427, 'num_write_kb': 43412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3146}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:47.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:47.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5145", 'reported_seq': 5220, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:44.332938+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:44.332938+0000', 'last_peered': '2024-09-11T04:58:44.332938+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:44.332938+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5145, 'log_dups_size': 0, 'ondisk_log_size': 5145, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42121808, 'num_objects': 5142, 'num_object_clones': 0, 'num_object_copies': 20568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5144, 'num_write_kb': 41142, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2922}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:47.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:47.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5335", 'reported_seq': 5397, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:46.697158+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:46.697158+0000', 'last_peered': '2024-09-11T04:58:46.697158+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:46.697158+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5335, 'log_dups_size': 0, 'ondisk_log_size': 5335, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43668640, 'num_objects': 5331, 'num_object_clones': 0, 'num_object_copies': 21324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5333, 'num_write_kb': 42653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3053}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5261", 'reported_seq': 5315, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:42.755309+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:42.755309+0000', 'last_peered': '2024-09-11T04:58:42.755309+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:42.755309+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5261, 'log_dups_size': 0, 'ondisk_log_size': 5261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43070624, 'num_objects': 5258, 'num_object_clones': 0, 'num_object_copies': 21032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5259, 'num_write_kb': 42069, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3008}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5306", 'reported_seq': 5347, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:42.790024+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:42.790024+0000', 'last_peered': '2024-09-11T04:58:42.790024+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:42.790024+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5306, 'log_dups_size': 0, 'ondisk_log_size': 5306, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43466752, 'num_objects': 5306, 'num_object_clones': 0, 'num_object_copies': 21224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5306, 'num_write_kb': 42448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2985}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5401", 'reported_seq': 5464, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:46.712915+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:46.712915+0000', 'last_peered': '2024-09-11T04:58:46.712915+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:46.712915+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5401, 'log_dups_size': 0, 'ondisk_log_size': 5401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44192928, 'num_objects': 5395, 'num_object_clones': 0, 'num_object_copies': 21580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5399, 'num_write_kb': 43172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3078}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5448", 'reported_seq': 5510, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:46.426170+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:46.426170+0000', 'last_peered': '2024-09-11T04:58:46.426170+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:46.426170+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5448, 'log_dups_size': 0, 'ondisk_log_size': 5448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44586144, 'num_objects': 5443, 'num_object_clones': 0, 'num_object_copies': 21772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5446, 'num_write_kb': 43556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3082}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5348", 'reported_seq': 5404, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:42.497267+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:42.497267+0000', 'last_peered': '2024-09-11T04:58:42.497267+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:42.497267+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5348, 'log_dups_size': 0, 'ondisk_log_size': 5348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43739456, 'num_objects': 5340, 'num_object_clones': 0, 'num_object_copies': 21360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5345, 'num_write_kb': 42737, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2946}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:47.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:47.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5318", 'reported_seq': 5395, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:44.283189+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:44.283189+0000', 'last_peered': '2024-09-11T04:58:44.283189+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:44.283189+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5318, 'log_dups_size': 0, 'ondisk_log_size': 5318, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43545760, 'num_objects': 5316, 'num_object_clones': 0, 'num_object_copies': 21264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5316, 'num_write_kb': 42526, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2998}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:47.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:47.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5316", 'reported_seq': 5372, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:42.698858+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:42.698858+0000', 'last_peered': '2024-09-11T04:58:42.698858+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:42.698858+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5316, 'log_dups_size': 0, 'ondisk_log_size': 5316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43503344, 'num_objects': 5311, 'num_object_clones': 0, 'num_object_copies': 21244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5313, 'num_write_kb': 42492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2988}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:47.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:47.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5162", 'reported_seq': 8249, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:42.767696+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:42.767696+0000', 'last_peered': '2024-09-11T04:58:42.767696+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:42.767696+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5162, 'log_dups_size': 0, 'ondisk_log_size': 5162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42277456, 'num_objects': 5161, 'num_object_clones': 0, 'num_object_copies': 20644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 711, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5161, 'num_write_kb': 41287, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1526, 'num_bytes_recovered': 12484608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:47.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:47.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5340", 'reported_seq': 5414, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:44.167077+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:44.167077+0000', 'last_peered': '2024-09-11T04:58:44.167077+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:44.167077+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5340, 'log_dups_size': 0, 'ondisk_log_size': 5340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43735632, 'num_objects': 5339, 'num_object_clones': 0, 'num_object_copies': 21356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5339, 'num_write_kb': 42711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:47.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:47.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5244", 'reported_seq': 5294, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:42.591687+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:42.591687+0000', 'last_peered': '2024-09-11T04:58:42.591687+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:42.591687+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5244, 'log_dups_size': 0, 'ondisk_log_size': 5244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42923168, 'num_objects': 5240, 'num_object_clones': 0, 'num_object_copies': 20960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5242, 'num_write_kb': 41932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:47.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:47.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5276", 'reported_seq': 5354, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:44.317451+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:44.317451+0000', 'last_peered': '2024-09-11T04:58:44.317451+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:44.317451+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5276, 'log_dups_size': 0, 'ondisk_log_size': 5276, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43193504, 'num_objects': 5273, 'num_object_clones': 0, 'num_object_copies': 21092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5274, 'num_write_kb': 42189, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2955}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:47.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:47.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5457", 'reported_seq': 5520, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:46.698785+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:46.698785+0000', 'last_peered': '2024-09-11T04:58:46.698785+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:46.698785+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5457, 'log_dups_size': 0, 'ondisk_log_size': 5457, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44659872, 'num_objects': 5452, 'num_object_clones': 0, 'num_object_copies': 21808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5455, 'num_write_kb': 43628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:47.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:48.684 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:48.685 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35358/339788 objects degraded (10.406%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":255,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84947,"data_bytes":695845056,"bytes_used":2311401472,"bytes_avail":384235655168,"bytes_total":386547056640,"degraded_objects":35358,"degraded_total":339788,"degraded_ratio":0.10405900149504985,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":78472,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"write_bytes_sec":2268210,"read_op_per_sec":0,"write_op_per_sec":276},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:48.710 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 185 20 86132 86112 3.63594 2.07812 0.0703207 0.0429599 2024-09-11T04:58:48.718 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:48.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:49.351 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:49.384 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-11T04:58:47.676467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":695845056,"num_objects":84947,"num_object_clones":0,"num_object_copies":339788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84974,"num_write_kb":679674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85001,"ondisk_log_size":85001,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2257228,"kb_used_data":1216204,"kb_used_omap":14,"kb_used_meta":1040945,"kb_avail":375230132,"statfs":{"total":386547056640,"available":384235655168,"internally_reserved":0,"allocated":1245392896,"data_stored":1244709656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1065928246},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,21,16,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27230208,"num_objects":3324,"num_object_clones":0,"num_object_copies":13296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005149"},"pg_stats":[{"pgid":"1.f","version":"31'5215","reported_seq":5255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.727394+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:42.727394+0000","last_peered":"2024-09-11T04:58:42.727394+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:42.727394+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42721280,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":41720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5429","reported_seq":5491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.756393+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:46.756393+0000","last_peered":"2024-09-11T04:58:46.756393+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:46.756393+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5429,"log_dups_size":0,"ondisk_log_size":5429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44438688,"num_objects":5425,"num_object_clones":0,"num_object_copies":21700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5427,"num_write_kb":43412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5145","reported_seq":5220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.332938+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:44.332938+0000","last_peered":"2024-09-11T04:58:44.332938+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:44.332938+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42121808,"num_objects":5142,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5144,"num_write_kb":41142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2922}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5335","reported_seq":5397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.697158+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:46.697158+0000","last_peered":"2024-09-11T04:58:46.697158+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:46.697158+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5333,"num_write_kb":42653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5261","reported_seq":5315,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.755309+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:42.755309+0000","last_peered":"2024-09-11T04:58:42.755309+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:42.755309+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43070624,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5306","reported_seq":5347,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.790024+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:42.790024+0000","last_peered":"2024-09-11T04:58:42.790024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:42.790024+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5306,"num_write_kb":42448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5401","reported_seq":5464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.712915+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:46.712915+0000","last_peered":"2024-09-11T04:58:46.712915+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:46.712915+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44192928,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3078}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5448","reported_seq":5510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.426170+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:46.426170+0000","last_peered":"2024-09-11T04:58:46.426170+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:46.426170+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586144,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3082}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5348","reported_seq":5404,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.497267+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:42.497267+0000","last_peered":"2024-09-11T04:58:42.497267+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:42.497267+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5348,"log_dups_size":0,"ondisk_log_size":5348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43739456,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5345,"num_write_kb":42737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5318","reported_seq":5395,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.283189+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:44.283189+0000","last_peered":"2024-09-11T04:58:44.283189+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:44.283189+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43545760,"num_objects":5316,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5316,"num_write_kb":42526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2998}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5316","reported_seq":5372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.698858+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:42.698858+0000","last_peered":"2024-09-11T04:58:42.698858+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:42.698858+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43503344,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5313,"num_write_kb":42492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5162","reported_seq":8249,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.767696+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:42.767696+0000","last_peered":"2024-09-11T04:58:42.767696+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:42.767696+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42277456,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":41287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1526,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4450}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5340","reported_seq":5414,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.167077+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:44.167077+0000","last_peered":"2024-09-11T04:58:44.167077+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:44.167077+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43735632,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":42711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3007}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5244","reported_seq":5294,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:42.591687+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:42.591687+0000","last_peered":"2024-09-11T04:58:42.591687+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:42.591687+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42923168,"num_objects":5240,"num_object_clones":0,"num_object_copies":20960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5242,"num_write_kb":41932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2953}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5276","reported_seq":5354,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:44.317451+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:44.317451+0000","last_peered":"2024-09-11T04:58:44.317451+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:44.317451+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43193504,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5274,"num_write_kb":42189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2955}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5457","reported_seq":5520,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.698785+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:46.698785+0000","last_peered":"2024-09-11T04:58:46.698785+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:46.698785+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44659872,"num_objects":5452,"num_object_clones":0,"num_object_copies":21808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5455,"num_write_kb":43628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":695845056,"num_objects":84947,"num_object_clones":0,"num_object_copies":339788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84974,"num_write_kb":679674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1245638656,"data_stored":1245638656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85001,"ondisk_log_size":85001,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608556,"kb_used_data":338076,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93763284,"statfs":{"total":96636764160,"available":96013602816,"internally_reserved":0,"allocated":346189824,"data_stored":346019014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":276950736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,11,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623512,"kb_used_data":339656,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93748328,"statfs":{"total":96636764160,"available":95998287872,"internally_reserved":0,"allocated":347807744,"data_stored":347636934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398584,"kb_used_data":196200,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93973256,"statfs":{"total":96636764160,"available":96228614144,"internally_reserved":0,"allocated":200908800,"data_stored":200737990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207220434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,9,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626576,"kb_used_data":342272,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93745264,"statfs":{"total":96636764160,"available":95995150336,"internally_reserved":0,"allocated":350486528,"data_stored":350315718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350715904,"data_stored":350715904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201121792,"data_stored":201121792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":347795456,"data_stored":347795456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346005504,"data_stored":346005504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:49.710 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 186 20 86411 86391 3.62811 2.17969 0.0909459 0.0430544 2024-09-11T04:58:50.710 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 187 20 86662 86642 3.61919 1.96094 0.0411929 0.0431533 2024-09-11T04:58:51.710 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 188 20 86959 86939 3.61228 2.32031 0.085488 0.0432421 2024-09-11T04:58:52.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:52.710 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 189 20 87236 87216 3.60462 2.16406 0.0334085 0.0433304 2024-09-11T04:58:53.053 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:53.054 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:53.077 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-11T04:58:51.677820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":704069824,"num_objects":85951,"num_object_clones":0,"num_object_copies":343804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85978,"num_write_kb":687706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":13017088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86005,"ondisk_log_size":86005,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2287472,"kb_used_data":1232560,"kb_used_omap":14,"kb_used_meta":1054833,"kb_avail":375199888,"statfs":{"total":386547056640,"available":384204685312,"internally_reserved":0,"allocated":1262141440,"data_stored":1261458200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1080149558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,8,6,12,11,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23691264,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2892,"num_write_kb":23136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004593"},"pg_stats":[{"pgid":"1.f","version":"31'5307","reported_seq":5349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:47.690831+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:47.690831+0000","last_peered":"2024-09-11T04:58:47.690831+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:47.690831+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5307,"log_dups_size":0,"ondisk_log_size":5307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43474944,"num_objects":5307,"num_object_clones":0,"num_object_copies":21228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5307,"num_write_kb":42456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5429","reported_seq":5491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.756393+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:46.756393+0000","last_peered":"2024-09-11T04:58:46.756393+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:46.756393+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5429,"log_dups_size":0,"ondisk_log_size":5429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44438688,"num_objects":5425,"num_object_clones":0,"num_object_copies":21700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5427,"num_write_kb":43412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5231","reported_seq":5306,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.253685+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:49.253685+0000","last_peered":"2024-09-11T04:58:49.253685+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:49.253685+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42826320,"num_objects":5228,"num_object_clones":0,"num_object_copies":20912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":41830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3008}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5335","reported_seq":5397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.697158+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:46.697158+0000","last_peered":"2024-09-11T04:58:46.697158+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:46.697158+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5333,"num_write_kb":42653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5368","reported_seq":5420,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:47.785072+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:47.785072+0000","last_peered":"2024-09-11T04:58:47.785072+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:47.785072+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43947168,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":42925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5396","reported_seq":5436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:47.854055+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:47.854055+0000","last_peered":"2024-09-11T04:58:47.854055+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:47.854055+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44204032,"num_objects":5396,"num_object_clones":0,"num_object_copies":21584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5396,"num_write_kb":43168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5401","reported_seq":5464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.712915+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:46.712915+0000","last_peered":"2024-09-11T04:58:46.712915+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:46.712915+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44192928,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3078}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5448","reported_seq":5510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.426170+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:46.426170+0000","last_peered":"2024-09-11T04:58:46.426170+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:46.426170+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44586144,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3082}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5450","reported_seq":5506,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:47.644669+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:47.644669+0000","last_peered":"2024-09-11T04:58:47.644669+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:47.644669+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5450,"log_dups_size":0,"ondisk_log_size":5450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44575040,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5447,"num_write_kb":43553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5387","reported_seq":5464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.318126+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:49.318126+0000","last_peered":"2024-09-11T04:58:49.318126+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:49.318126+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44111008,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5385,"num_write_kb":43078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5399","reported_seq":5454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:47.750808+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:47.750808+0000","last_peered":"2024-09-11T04:58:47.750808+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:47.750808+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5399,"log_dups_size":0,"ondisk_log_size":5399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44183280,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5396,"num_write_kb":43156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3071}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5246","reported_seq":8463,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:47.870090+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:47.870090+0000","last_peered":"2024-09-11T04:58:47.870090+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:47.870090+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5246,"log_dups_size":0,"ondisk_log_size":5246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42965584,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5245,"num_write_kb":41959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1591,"num_bytes_recovered":13017088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4599}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5437","reported_seq":5510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.400294+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:49.400294+0000","last_peered":"2024-09-11T04:58:49.400294+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:49.400294+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44530256,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5436,"num_write_kb":43487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5354","reported_seq":5404,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:47.763220+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:47.763220+0000","last_peered":"2024-09-11T04:58:47.763220+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:47.763220+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5354,"log_dups_size":0,"ondisk_log_size":5354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43824288,"num_objects":5350,"num_object_clones":0,"num_object_copies":21400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5352,"num_write_kb":42812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3063}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5360","reported_seq":5436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.311501+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:49.311501+0000","last_peered":"2024-09-11T04:58:49.311501+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:49.311501+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43881632,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":42861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3039}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5457","reported_seq":5520,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:46.698785+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:46.698785+0000","last_peered":"2024-09-11T04:58:46.698785+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:46.698785+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44659872,"num_objects":5452,"num_object_clones":0,"num_object_copies":21808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5455,"num_write_kb":43628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":704069824,"num_objects":85951,"num_object_clones":0,"num_object_copies":343804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85978,"num_write_kb":687706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":13017088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1263505408,"data_stored":1263505408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86005,"ondisk_log_size":86005,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635588,"kb_used_data":342900,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93736252,"statfs":{"total":96636764160,"available":95985922048,"internally_reserved":0,"allocated":351129600,"data_stored":350958790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":299691728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,5,6,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633324,"kb_used_data":345308,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93738516,"statfs":{"total":96636764160,"available":95988240384,"internally_reserved":0,"allocated":353595392,"data_stored":353424582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391984,"kb_used_data":202080,"kb_used_omap":4,"kb_used_meta":189883,"kb_avail":93979856,"statfs":{"total":96636764160,"available":96235372544,"internally_reserved":0,"allocated":206929920,"data_stored":206759110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":194440914},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626576,"kb_used_data":342272,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93745264,"statfs":{"total":96636764160,"available":95995150336,"internally_reserved":0,"allocated":350486528,"data_stored":350315718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350715904,"data_stored":350715904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":207290368,"data_stored":207290368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353562624,"data_stored":353562624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351936512,"data_stored":351936512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:53.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:53.711 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 190 20 87504 87484 3.59665 2.09375 0.0871326 0.0434294 2024-09-11T04:58:53.729 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:53.732 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:53.770 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-11T04:58:53.678816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":712417472,"num_objects":86970,"num_object_clones":0,"num_object_copies":347880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86997,"num_write_kb":695858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87024,"ondisk_log_size":87024,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2300088,"kb_used_data":1249332,"kb_used_omap":14,"kb_used_meta":1050673,"kb_avail":375187272,"statfs":{"total":386547056640,"available":384191766528,"internally_reserved":0,"allocated":1279315968,"data_stored":1278632728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1075889718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,5,8,10,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32038912,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004853"},"pg_stats":[{"pgid":"1.f","version":"31'5407","reported_seq":5449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.834228+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:52.834228+0000","last_peered":"2024-09-11T04:58:52.834228+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:52.834228+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44294144,"num_objects":5407,"num_object_clones":0,"num_object_copies":21628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5407,"num_write_kb":43256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3095}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5528","reported_seq":5588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.790099+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:51.790099+0000","last_peered":"2024-09-11T04:58:51.790099+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:51.790099+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249696,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5526,"num_write_kb":44204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5231","reported_seq":5306,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.253685+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:49.253685+0000","last_peered":"2024-09-11T04:58:49.253685+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:49.253685+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42826320,"num_objects":5228,"num_object_clones":0,"num_object_copies":20912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":41830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3008}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5393","reported_seq":5456,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.757551+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:51.757551+0000","last_peered":"2024-09-11T04:58:51.757551+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:51.757551+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143776,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5452","reported_seq":5505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.815990+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:52.815990+0000","last_peered":"2024-09-11T04:58:52.815990+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:52.815990+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5452,"log_dups_size":0,"ondisk_log_size":5452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635296,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5450,"num_write_kb":43597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5491","reported_seq":5532,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.871051+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:52.871051+0000","last_peered":"2024-09-11T04:58:52.871051+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:52.871051+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44982272,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":43928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5487","reported_seq":5550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.751551+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:51.751551+0000","last_peered":"2024-09-11T04:58:51.751551+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:51.751551+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44897440,"num_objects":5481,"num_object_clones":0,"num_object_copies":21924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3164}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5528","reported_seq":5590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.575422+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:51.575422+0000","last_peered":"2024-09-11T04:58:51.575422+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:51.575422+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45241504,"num_objects":5523,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5526,"num_write_kb":44196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5535","reported_seq":5590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.945148+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:52.945148+0000","last_peered":"2024-09-11T04:58:52.945148+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:52.945148+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5535,"log_dups_size":0,"ondisk_log_size":5535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45271360,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5387","reported_seq":5464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.318126+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:49.318126+0000","last_peered":"2024-09-11T04:58:49.318126+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:49.318126+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44111008,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5385,"num_write_kb":43078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5489","reported_seq":5545,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.898278+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:52.898278+0000","last_peered":"2024-09-11T04:58:52.898278+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:52.898278+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44920560,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":43876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5339","reported_seq":8748,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.981533+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:52.981533+0000","last_peered":"2024-09-11T04:58:52.981533+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:52.981533+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5339,"log_dups_size":0,"ondisk_log_size":5339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43727440,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4789}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5437","reported_seq":5510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.400294+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:49.400294+0000","last_peered":"2024-09-11T04:58:49.400294+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:49.400294+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44530256,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5436,"num_write_kb":43487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5432","reported_seq":5481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.941751+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:52.941751+0000","last_peered":"2024-09-11T04:58:52.941751+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:52.941751+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44463264,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":43436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5360","reported_seq":5436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.311501+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:49.311501+0000","last_peered":"2024-09-11T04:58:49.311501+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:49.311501+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43881632,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":42861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3039}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5528","reported_seq":5591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.754000+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:51.754000+0000","last_peered":"2024-09-11T04:58:51.754000+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:51.754000+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45241504,"num_objects":5523,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5526,"num_write_kb":44196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":712417472,"num_objects":86970,"num_object_clones":0,"num_object_copies":347880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86997,"num_write_kb":695858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280806912,"data_stored":1280806912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87024,"ondisk_log_size":87024,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628524,"kb_used_data":348316,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93743316,"statfs":{"total":96636764160,"available":95993155584,"internally_reserved":0,"allocated":356675584,"data_stored":356504774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":286912208},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,6,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633324,"kb_used_data":345308,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93738516,"statfs":{"total":96636764160,"available":95988240384,"internally_reserved":0,"allocated":353595392,"data_stored":353424582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402008,"kb_used_data":207940,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93969832,"statfs":{"total":96636764160,"available":96225107968,"internally_reserved":0,"allocated":212930560,"data_stored":212759750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198700754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636232,"kb_used_data":347768,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93735608,"statfs":{"total":96636764160,"available":95985262592,"internally_reserved":0,"allocated":356114432,"data_stored":355943622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356311040,"data_stored":356311040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213348352,"data_stored":213348352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353562624,"data_stored":353562624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357584896,"data_stored":357584896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:53.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:53.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5407", 'reported_seq': 5449, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.834228+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:52.834228+0000', 'last_peered': '2024-09-11T04:58:52.834228+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:52.834228+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5407, 'log_dups_size': 0, 'ondisk_log_size': 5407, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44294144, 'num_objects': 5407, 'num_object_clones': 0, 'num_object_copies': 21628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5407, 'num_write_kb': 43256, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3095}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:53.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:53.772 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5528", 'reported_seq': 5588, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:51.790099+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:51.790099+0000', 'last_peered': '2024-09-11T04:58:51.790099+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:51.790099+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5528, 'log_dups_size': 0, 'ondisk_log_size': 5528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45249696, 'num_objects': 5524, 'num_object_clones': 0, 'num_object_copies': 22096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5526, 'num_write_kb': 44204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3245}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:53.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:53.772 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5231", 'reported_seq': 5306, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:49.253685+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:49.253685+0000', 'last_peered': '2024-09-11T04:58:49.253685+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:49.253685+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5231, 'log_dups_size': 0, 'ondisk_log_size': 5231, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42826320, 'num_objects': 5228, 'num_object_clones': 0, 'num_object_copies': 20912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5228, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5230, 'num_write_kb': 41830, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3008}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:53.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:53.772 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5393", 'reported_seq': 5456, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:51.757551+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:51.757551+0000', 'last_peered': '2024-09-11T04:58:51.757551+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:51.757551+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5393, 'log_dups_size': 0, 'ondisk_log_size': 5393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44143776, 'num_objects': 5389, 'num_object_clones': 0, 'num_object_copies': 21556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5391, 'num_write_kb': 43117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:53.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:53.772 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5452", 'reported_seq': 5505, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.815990+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:52.815990+0000', 'last_peered': '2024-09-11T04:58:52.815990+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:52.815990+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5452, 'log_dups_size': 0, 'ondisk_log_size': 5452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44635296, 'num_objects': 5449, 'num_object_clones': 0, 'num_object_copies': 21796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5450, 'num_write_kb': 43597, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3199}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5491", 'reported_seq': 5532, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.871051+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:52.871051+0000', 'last_peered': '2024-09-11T04:58:52.871051+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:52.871051+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5491, 'log_dups_size': 0, 'ondisk_log_size': 5491, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44982272, 'num_objects': 5491, 'num_object_clones': 0, 'num_object_copies': 21964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5491, 'num_write_kb': 43928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3170}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5487", 'reported_seq': 5550, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:51.751551+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:51.751551+0000', 'last_peered': '2024-09-11T04:58:51.751551+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:51.751551+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5487, 'log_dups_size': 0, 'ondisk_log_size': 5487, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44897440, 'num_objects': 5481, 'num_object_clones': 0, 'num_object_copies': 21924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5485, 'num_write_kb': 43860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3164}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5528", 'reported_seq': 5590, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:51.575422+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:51.575422+0000', 'last_peered': '2024-09-11T04:58:51.575422+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:51.575422+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5528, 'log_dups_size': 0, 'ondisk_log_size': 5528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45241504, 'num_objects': 5523, 'num_object_clones': 0, 'num_object_copies': 22092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5526, 'num_write_kb': 44196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5535", 'reported_seq': 5590, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.945148+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:52.945148+0000', 'last_peered': '2024-09-11T04:58:52.945148+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:52.945148+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5535, 'log_dups_size': 0, 'ondisk_log_size': 5535, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45271360, 'num_objects': 5527, 'num_object_clones': 0, 'num_object_copies': 22108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5532, 'num_write_kb': 44233, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3133}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:53.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5387", 'reported_seq': 5464, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:49.318126+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:49.318126+0000', 'last_peered': '2024-09-11T04:58:49.318126+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:49.318126+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5387, 'log_dups_size': 0, 'ondisk_log_size': 5387, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44111008, 'num_objects': 5385, 'num_object_clones': 0, 'num_object_copies': 21540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5385, 'num_write_kb': 43078, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3067}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5489", 'reported_seq': 5545, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.898278+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:52.898278+0000', 'last_peered': '2024-09-11T04:58:52.898278+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:52.898278+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5489, 'log_dups_size': 0, 'ondisk_log_size': 5489, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44920560, 'num_objects': 5484, 'num_object_clones': 0, 'num_object_copies': 21936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5486, 'num_write_kb': 43876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5339", 'reported_seq': 8748, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.981533+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:52.981533+0000', 'last_peered': '2024-09-11T04:58:52.981533+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:52.981533+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5339, 'log_dups_size': 0, 'ondisk_log_size': 5339, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43727440, 'num_objects': 5338, 'num_object_clones': 0, 'num_object_copies': 21352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 549, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5338, 'num_write_kb': 42703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1688, 'num_bytes_recovered': 13811712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4789}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5437", 'reported_seq': 5510, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:49.400294+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:49.400294+0000', 'last_peered': '2024-09-11T04:58:49.400294+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:49.400294+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5437, 'log_dups_size': 0, 'ondisk_log_size': 5437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44530256, 'num_objects': 5436, 'num_object_clones': 0, 'num_object_copies': 21744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5436, 'num_write_kb': 43487, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5432", 'reported_seq': 5481, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.941751+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:52.941751+0000', 'last_peered': '2024-09-11T04:58:52.941751+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:52.941751+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5432, 'log_dups_size': 0, 'ondisk_log_size': 5432, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44463264, 'num_objects': 5428, 'num_object_clones': 0, 'num_object_copies': 21712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5430, 'num_write_kb': 43436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:53.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5360", 'reported_seq': 5436, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:49.311501+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:49.311501+0000', 'last_peered': '2024-09-11T04:58:49.311501+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:49.311501+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5360, 'log_dups_size': 0, 'ondisk_log_size': 5360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43881632, 'num_objects': 5357, 'num_object_clones': 0, 'num_object_copies': 21428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5358, 'num_write_kb': 42861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3039}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:53.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:53.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5528", 'reported_seq': 5591, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:51.754000+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:51.754000+0000', 'last_peered': '2024-09-11T04:58:51.754000+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:51.754000+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5528, 'log_dups_size': 0, 'ondisk_log_size': 5528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45241504, 'num_objects': 5523, 'num_object_clones': 0, 'num_object_copies': 22092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5526, 'num_write_kb': 44196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3212}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:53.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:54.531 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:54.532 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35196/347880 objects degraded (10.117%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":261,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86970,"data_bytes":712417472,"bytes_used":2355290112,"bytes_avail":384191766528,"bytes_total":386547056640,"degraded_objects":35196,"degraded_total":347880,"degraded_ratio":0.10117281821317697,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":149443,"recovering_keys_per_sec":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"write_bytes_sec":2668829,"read_op_per_sec":0,"write_op_per_sec":325},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:58:54.554 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:58:54.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:54.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 191 20 87772 87752 3.58878 2.09375 0.0791275 0.0435264 2024-09-11T04:58:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:55.165 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:55.191 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-11T04:58:53.678816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":712417472,"num_objects":86970,"num_object_clones":0,"num_object_copies":347880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86997,"num_write_kb":695858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87024,"ondisk_log_size":87024,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2300088,"kb_used_data":1249332,"kb_used_omap":14,"kb_used_meta":1050673,"kb_avail":375187272,"statfs":{"total":386547056640,"available":384191766528,"internally_reserved":0,"allocated":1279315968,"data_stored":1278632728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1075889718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,5,8,10,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32038912,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004853"},"pg_stats":[{"pgid":"1.f","version":"31'5407","reported_seq":5449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.834228+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:52.834228+0000","last_peered":"2024-09-11T04:58:52.834228+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:52.834228+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44294144,"num_objects":5407,"num_object_clones":0,"num_object_copies":21628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5407,"num_write_kb":43256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3095}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5528","reported_seq":5588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.790099+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:51.790099+0000","last_peered":"2024-09-11T04:58:51.790099+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:51.790099+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249696,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5526,"num_write_kb":44204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5231","reported_seq":5306,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.253685+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:49.253685+0000","last_peered":"2024-09-11T04:58:49.253685+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:49.253685+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42826320,"num_objects":5228,"num_object_clones":0,"num_object_copies":20912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":41830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3008}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5393","reported_seq":5456,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.757551+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:51.757551+0000","last_peered":"2024-09-11T04:58:51.757551+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:51.757551+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143776,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5452","reported_seq":5505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.815990+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:52.815990+0000","last_peered":"2024-09-11T04:58:52.815990+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:52.815990+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5452,"log_dups_size":0,"ondisk_log_size":5452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635296,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5450,"num_write_kb":43597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5491","reported_seq":5532,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.871051+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:52.871051+0000","last_peered":"2024-09-11T04:58:52.871051+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:52.871051+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44982272,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":43928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5487","reported_seq":5550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.751551+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:51.751551+0000","last_peered":"2024-09-11T04:58:51.751551+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:51.751551+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44897440,"num_objects":5481,"num_object_clones":0,"num_object_copies":21924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3164}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5528","reported_seq":5590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.575422+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:51.575422+0000","last_peered":"2024-09-11T04:58:51.575422+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:51.575422+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45241504,"num_objects":5523,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5526,"num_write_kb":44196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5535","reported_seq":5590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.945148+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:52.945148+0000","last_peered":"2024-09-11T04:58:52.945148+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:52.945148+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5535,"log_dups_size":0,"ondisk_log_size":5535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45271360,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5387","reported_seq":5464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.318126+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:49.318126+0000","last_peered":"2024-09-11T04:58:49.318126+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:49.318126+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44111008,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5385,"num_write_kb":43078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5489","reported_seq":5545,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.898278+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:52.898278+0000","last_peered":"2024-09-11T04:58:52.898278+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:52.898278+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44920560,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":43876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5339","reported_seq":8748,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.981533+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:52.981533+0000","last_peered":"2024-09-11T04:58:52.981533+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:52.981533+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5339,"log_dups_size":0,"ondisk_log_size":5339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43727440,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4789}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5437","reported_seq":5510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.400294+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:49.400294+0000","last_peered":"2024-09-11T04:58:49.400294+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:49.400294+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44530256,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5436,"num_write_kb":43487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5432","reported_seq":5481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.941751+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:52.941751+0000","last_peered":"2024-09-11T04:58:52.941751+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:52.941751+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44463264,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":43436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5360","reported_seq":5436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:49.311501+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:49.311501+0000","last_peered":"2024-09-11T04:58:49.311501+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:49.311501+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43881632,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":42861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3039}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5528","reported_seq":5591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:51.754000+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:51.754000+0000","last_peered":"2024-09-11T04:58:51.754000+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:51.754000+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45241504,"num_objects":5523,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5526,"num_write_kb":44196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":712417472,"num_objects":86970,"num_object_clones":0,"num_object_copies":347880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86997,"num_write_kb":695858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280806912,"data_stored":1280806912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87024,"ondisk_log_size":87024,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628524,"kb_used_data":348316,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93743316,"statfs":{"total":96636764160,"available":95993155584,"internally_reserved":0,"allocated":356675584,"data_stored":356504774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":286912208},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,6,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633324,"kb_used_data":345308,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93738516,"statfs":{"total":96636764160,"available":95988240384,"internally_reserved":0,"allocated":353595392,"data_stored":353424582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402008,"kb_used_data":207940,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93969832,"statfs":{"total":96636764160,"available":96225107968,"internally_reserved":0,"allocated":212930560,"data_stored":212759750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198700754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636232,"kb_used_data":347768,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93735608,"statfs":{"total":96636764160,"available":95985262592,"internally_reserved":0,"allocated":356114432,"data_stored":355943622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356311040,"data_stored":356311040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213348352,"data_stored":213348352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353562624,"data_stored":353562624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357584896,"data_stored":357584896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:55.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 192 20 88049 88029 3.58136 2.16406 0.0908642 0.0436145 2024-09-11T04:58:56.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 193 20 88335 88315 3.57438 2.23438 0.0521796 0.0437026 2024-09-11T04:58:57.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 194 20 88624 88604 3.56759 2.25781 0.0410669 0.043785 2024-09-11T04:58:58.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:58.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 195 20 88890 88870 3.55995 2.07812 0.0709693 0.0438732 2024-09-11T04:58:58.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:58.836 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:58.859 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-11T04:58:57.680501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":718676160,"num_objects":87734,"num_object_clones":0,"num_object_copies":350936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87761,"num_write_kb":701970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87788,"ondisk_log_size":87788,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2337528,"kb_used_data":1260404,"kb_used_omap":14,"kb_used_meta":1077041,"kb_avail":375149832,"statfs":{"total":386547056640,"available":384153427968,"internally_reserved":0,"allocated":1290653696,"data_stored":1289970456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1102890550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,18,18,11,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26460160,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":25840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004709"},"pg_stats":[{"pgid":"1.f","version":"31'5407","reported_seq":5449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.834228+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:52.834228+0000","last_peered":"2024-09-11T04:58:52.834228+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:52.834228+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44294144,"num_objects":5407,"num_object_clones":0,"num_object_copies":21628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5407,"num_write_kb":43256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3095}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5608","reported_seq":5670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.765634+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:56.765634+0000","last_peered":"2024-09-11T04:58:56.765634+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:56.765634+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905056,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":44844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5311","reported_seq":5386,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:54.452042+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:54.452042+0000","last_peered":"2024-09-11T04:58:54.452042+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:54.452042+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43481680,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5310,"num_write_kb":42470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3088}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5465","reported_seq":5528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.798157+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:56.798157+0000","last_peered":"2024-09-11T04:58:56.798157+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:56.798157+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44733600,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5463,"num_write_kb":43693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5452","reported_seq":5505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.815990+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:52.815990+0000","last_peered":"2024-09-11T04:58:52.815990+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:52.815990+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5452,"log_dups_size":0,"ondisk_log_size":5452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635296,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5450,"num_write_kb":43597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5491","reported_seq":5532,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.871051+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:52.871051+0000","last_peered":"2024-09-11T04:58:52.871051+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:52.871051+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44982272,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":43928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5575","reported_seq":5638,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.810855+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:56.810855+0000","last_peered":"2024-09-11T04:58:56.810855+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:56.810855+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45618336,"num_objects":5569,"num_object_clones":0,"num_object_copies":22276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":44564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3252}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5602","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.840269+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:56.840269+0000","last_peered":"2024-09-11T04:58:56.840269+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:56.840269+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45847712,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5535","reported_seq":5590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.945148+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:52.945148+0000","last_peered":"2024-09-11T04:58:52.945148+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:52.945148+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5535,"log_dups_size":0,"ondisk_log_size":5535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45271360,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5493","reported_seq":5570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:54.235041+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:54.235041+0000","last_peered":"2024-09-11T04:58:54.235041+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:54.235041+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44979360,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":43926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3173}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5489","reported_seq":5545,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.898278+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:52.898278+0000","last_peered":"2024-09-11T04:58:52.898278+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:52.898278+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44920560,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":43876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5339","reported_seq":8748,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.981533+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:52.981533+0000","last_peered":"2024-09-11T04:58:52.981533+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:52.981533+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5339,"log_dups_size":0,"ondisk_log_size":5339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43727440,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4789}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5516","reported_seq":5589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:54.325716+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:54.325716+0000","last_peered":"2024-09-11T04:58:54.325716+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:54.325716+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5516,"log_dups_size":0,"ondisk_log_size":5516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45177424,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5515,"num_write_kb":44119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5432","reported_seq":5481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.941751+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:52.941751+0000","last_peered":"2024-09-11T04:58:52.941751+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:52.941751+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44463264,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":43436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5463","reported_seq":5540,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:54.369864+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:54.369864+0000","last_peered":"2024-09-11T04:58:54.369864+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:54.369864+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5463,"log_dups_size":0,"ondisk_log_size":5463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44725408,"num_objects":5460,"num_object_clones":0,"num_object_copies":21840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5461,"num_write_kb":43685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3142}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5610","reported_seq":5673,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.765449+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:56.765449+0000","last_peered":"2024-09-11T04:58:56.765449+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:56.765449+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45913248,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":44852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3294}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":718676160,"num_objects":87734,"num_object_clones":0,"num_object_copies":350936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87761,"num_write_kb":701970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292136448,"data_stored":1292136448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87788,"ondisk_log_size":87788,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628524,"kb_used_data":348316,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93743316,"statfs":{"total":96636764160,"available":95993155584,"internally_reserved":0,"allocated":356675584,"data_stored":356504774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":286912208},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,6,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642940,"kb_used_data":350764,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93728900,"statfs":{"total":96636764160,"available":95978393600,"internally_reserved":0,"allocated":359182336,"data_stored":359011526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402008,"kb_used_data":207940,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93969832,"statfs":{"total":96636764160,"available":96225107968,"internally_reserved":0,"allocated":212930560,"data_stored":212759750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198700754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664056,"kb_used_data":353384,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93707784,"statfs":{"total":96636764160,"available":95956770816,"internally_reserved":0,"allocated":361865216,"data_stored":361694406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,12,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361963520,"data_stored":361963520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213348352,"data_stored":213348352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359239680,"data_stored":359239680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357584896,"data_stored":357584896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:58.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:58:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:58:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:58:59.529 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-11T04:58:57.680501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":718676160,"num_objects":87734,"num_object_clones":0,"num_object_copies":350936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87761,"num_write_kb":701970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87788,"ondisk_log_size":87788,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2337528,"kb_used_data":1260404,"kb_used_omap":14,"kb_used_meta":1077041,"kb_avail":375149832,"statfs":{"total":386547056640,"available":384153427968,"internally_reserved":0,"allocated":1290653696,"data_stored":1289970456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1102890550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,18,18,11,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26460160,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":25840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004709"},"pg_stats":[{"pgid":"1.f","version":"31'5407","reported_seq":5449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.834228+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:52.834228+0000","last_peered":"2024-09-11T04:58:52.834228+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:52.834228+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44294144,"num_objects":5407,"num_object_clones":0,"num_object_copies":21628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5407,"num_write_kb":43256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3095}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5608","reported_seq":5670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.765634+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:56.765634+0000","last_peered":"2024-09-11T04:58:56.765634+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:56.765634+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905056,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":44844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5311","reported_seq":5386,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:54.452042+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:54.452042+0000","last_peered":"2024-09-11T04:58:54.452042+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:54.452042+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43481680,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5310,"num_write_kb":42470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3088}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5465","reported_seq":5528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.798157+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:56.798157+0000","last_peered":"2024-09-11T04:58:56.798157+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:56.798157+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44733600,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5463,"num_write_kb":43693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5452","reported_seq":5505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.815990+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:52.815990+0000","last_peered":"2024-09-11T04:58:52.815990+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:52.815990+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5452,"log_dups_size":0,"ondisk_log_size":5452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635296,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5450,"num_write_kb":43597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5491","reported_seq":5532,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.871051+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:52.871051+0000","last_peered":"2024-09-11T04:58:52.871051+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:52.871051+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44982272,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":43928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5575","reported_seq":5638,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.810855+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:56.810855+0000","last_peered":"2024-09-11T04:58:56.810855+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:56.810855+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45618336,"num_objects":5569,"num_object_clones":0,"num_object_copies":22276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":44564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3252}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5602","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.840269+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:56.840269+0000","last_peered":"2024-09-11T04:58:56.840269+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:56.840269+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45847712,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5535","reported_seq":5590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.945148+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:52.945148+0000","last_peered":"2024-09-11T04:58:52.945148+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:52.945148+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5535,"log_dups_size":0,"ondisk_log_size":5535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45271360,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5493","reported_seq":5570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:54.235041+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:54.235041+0000","last_peered":"2024-09-11T04:58:54.235041+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:54.235041+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44979360,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":43926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3173}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5489","reported_seq":5545,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.898278+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:52.898278+0000","last_peered":"2024-09-11T04:58:52.898278+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:52.898278+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44920560,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":43876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5339","reported_seq":8748,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.981533+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:52.981533+0000","last_peered":"2024-09-11T04:58:52.981533+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:52.981533+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5339,"log_dups_size":0,"ondisk_log_size":5339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43727440,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4789}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5516","reported_seq":5589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:54.325716+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:54.325716+0000","last_peered":"2024-09-11T04:58:54.325716+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:54.325716+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5516,"log_dups_size":0,"ondisk_log_size":5516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45177424,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5515,"num_write_kb":44119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5432","reported_seq":5481,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:52.941751+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:52.941751+0000","last_peered":"2024-09-11T04:58:52.941751+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:52.941751+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44463264,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":43436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5463","reported_seq":5540,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:54.369864+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:54.369864+0000","last_peered":"2024-09-11T04:58:54.369864+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:54.369864+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5463,"log_dups_size":0,"ondisk_log_size":5463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44725408,"num_objects":5460,"num_object_clones":0,"num_object_copies":21840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5461,"num_write_kb":43685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3142}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5610","reported_seq":5673,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.765449+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:56.765449+0000","last_peered":"2024-09-11T04:58:56.765449+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:56.765449+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45913248,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":44852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3294}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":718676160,"num_objects":87734,"num_object_clones":0,"num_object_copies":350936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87761,"num_write_kb":701970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":13811712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292136448,"data_stored":1292136448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87788,"ondisk_log_size":87788,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628524,"kb_used_data":348316,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93743316,"statfs":{"total":96636764160,"available":95993155584,"internally_reserved":0,"allocated":356675584,"data_stored":356504774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":286912208},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,6,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642940,"kb_used_data":350764,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93728900,"statfs":{"total":96636764160,"available":95978393600,"internally_reserved":0,"allocated":359182336,"data_stored":359011526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402008,"kb_used_data":207940,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93969832,"statfs":{"total":96636764160,"available":96225107968,"internally_reserved":0,"allocated":212930560,"data_stored":212759750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":198700754},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664056,"kb_used_data":353384,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93707784,"statfs":{"total":96636764160,"available":95956770816,"internally_reserved":0,"allocated":361865216,"data_stored":361694406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,12,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361963520,"data_stored":361963520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213348352,"data_stored":213348352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359239680,"data_stored":359239680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357584896,"data_stored":357584896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:58:59.530 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:58:59.530 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5407", 'reported_seq': 5449, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.834228+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:58:52.834228+0000', 'last_peered': '2024-09-11T04:58:52.834228+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:58:52.834228+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5407, 'log_dups_size': 0, 'ondisk_log_size': 5407, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44294144, 'num_objects': 5407, 'num_object_clones': 0, 'num_object_copies': 21628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5407, 'num_write_kb': 43256, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3095}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5608", 'reported_seq': 5670, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:56.765634+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:58:56.765634+0000', 'last_peered': '2024-09-11T04:58:56.765634+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:58:56.765634+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5608, 'log_dups_size': 0, 'ondisk_log_size': 5608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45905056, 'num_objects': 5604, 'num_object_clones': 0, 'num_object_copies': 22416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5606, 'num_write_kb': 44844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3325}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5311", 'reported_seq': 5386, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:54.452042+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:54.452042+0000', 'last_peered': '2024-09-11T04:58:54.452042+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:54.452042+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5311, 'log_dups_size': 0, 'ondisk_log_size': 5311, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43481680, 'num_objects': 5308, 'num_object_clones': 0, 'num_object_copies': 21232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5310, 'num_write_kb': 42470, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3088}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5465", 'reported_seq': 5528, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:56.798157+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:58:56.798157+0000', 'last_peered': '2024-09-11T04:58:56.798157+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:58:56.798157+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5465, 'log_dups_size': 0, 'ondisk_log_size': 5465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44733600, 'num_objects': 5461, 'num_object_clones': 0, 'num_object_copies': 21844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5463, 'num_write_kb': 43693, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5452", 'reported_seq': 5505, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.815990+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:58:52.815990+0000', 'last_peered': '2024-09-11T04:58:52.815990+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:58:52.815990+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5452, 'log_dups_size': 0, 'ondisk_log_size': 5452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44635296, 'num_objects': 5449, 'num_object_clones': 0, 'num_object_copies': 21796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5450, 'num_write_kb': 43597, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3199}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:58:59.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5491", 'reported_seq': 5532, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.871051+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:58:52.871051+0000', 'last_peered': '2024-09-11T04:58:52.871051+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:58:52.871051+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5491, 'log_dups_size': 0, 'ondisk_log_size': 5491, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44982272, 'num_objects': 5491, 'num_object_clones': 0, 'num_object_copies': 21964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5491, 'num_write_kb': 43928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3170}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5575", 'reported_seq': 5638, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:56.810855+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:58:56.810855+0000', 'last_peered': '2024-09-11T04:58:56.810855+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:58:56.810855+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5575, 'log_dups_size': 0, 'ondisk_log_size': 5575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45618336, 'num_objects': 5569, 'num_object_clones': 0, 'num_object_copies': 22276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5573, 'num_write_kb': 44564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3252}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5602", 'reported_seq': 5662, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:56.840269+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:58:56.840269+0000', 'last_peered': '2024-09-11T04:58:56.840269+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:58:56.840269+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5602, 'log_dups_size': 0, 'ondisk_log_size': 5602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45847712, 'num_objects': 5597, 'num_object_clones': 0, 'num_object_copies': 22388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5600, 'num_write_kb': 44788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3236}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5535", 'reported_seq': 5590, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.945148+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:58:52.945148+0000', 'last_peered': '2024-09-11T04:58:52.945148+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:58:52.945148+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5535, 'log_dups_size': 0, 'ondisk_log_size': 5535, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45271360, 'num_objects': 5527, 'num_object_clones': 0, 'num_object_copies': 22108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5532, 'num_write_kb': 44233, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3133}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5493", 'reported_seq': 5570, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:54.235041+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:54.235041+0000', 'last_peered': '2024-09-11T04:58:54.235041+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:54.235041+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5493, 'log_dups_size': 0, 'ondisk_log_size': 5493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44979360, 'num_objects': 5491, 'num_object_clones': 0, 'num_object_copies': 21964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5491, 'num_write_kb': 43926, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3173}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:58:59.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5489", 'reported_seq': 5545, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.898278+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:58:52.898278+0000', 'last_peered': '2024-09-11T04:58:52.898278+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:58:52.898278+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5489, 'log_dups_size': 0, 'ondisk_log_size': 5489, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44920560, 'num_objects': 5484, 'num_object_clones': 0, 'num_object_copies': 21936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5486, 'num_write_kb': 43876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5339", 'reported_seq': 8748, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.981533+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:58:52.981533+0000', 'last_peered': '2024-09-11T04:58:52.981533+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:58:52.981533+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5339, 'log_dups_size': 0, 'ondisk_log_size': 5339, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43727440, 'num_objects': 5338, 'num_object_clones': 0, 'num_object_copies': 21352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 549, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5338, 'num_write_kb': 42703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1688, 'num_bytes_recovered': 13811712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4789}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5516", 'reported_seq': 5589, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:54.325716+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:54.325716+0000', 'last_peered': '2024-09-11T04:58:54.325716+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:54.325716+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5516, 'log_dups_size': 0, 'ondisk_log_size': 5516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45177424, 'num_objects': 5515, 'num_object_clones': 0, 'num_object_copies': 22060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5515, 'num_write_kb': 44119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5432", 'reported_seq': 5481, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:52.941751+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:58:52.941751+0000', 'last_peered': '2024-09-11T04:58:52.941751+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:58:52.941751+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5432, 'log_dups_size': 0, 'ondisk_log_size': 5432, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44463264, 'num_objects': 5428, 'num_object_clones': 0, 'num_object_copies': 21712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5430, 'num_write_kb': 43436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5463", 'reported_seq': 5540, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:54.369864+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:54.369864+0000', 'last_peered': '2024-09-11T04:58:54.369864+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:54.369864+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5463, 'log_dups_size': 0, 'ondisk_log_size': 5463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44725408, 'num_objects': 5460, 'num_object_clones': 0, 'num_object_copies': 21840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5461, 'num_write_kb': 43685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3142}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:58:59.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5610", 'reported_seq': 5673, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:56.765449+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:58:56.765449+0000', 'last_peered': '2024-09-11T04:58:56.765449+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:58:56.765449+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5610, 'log_dups_size': 0, 'ondisk_log_size': 5610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45913248, 'num_objects': 5605, 'num_object_clones': 0, 'num_object_copies': 22420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5608, 'num_write_kb': 44852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3294}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:58:59.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:58:59.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 196 20 89145 89125 3.55195 1.99219 0.0720568 0.0439744 2024-09-11T04:59:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:00.313 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35139/354840 objects degraded (9.903%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":267,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88710,"data_bytes":726671552,"bytes_used":2518573056,"bytes_avail":384028483584,"bytes_total":386547056640,"degraded_objects":35139,"degraded_total":354840,"degraded_ratio":0.099027730808251613,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":149444,"recovering_keys_per_sec":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"write_bytes_sec":2567854,"read_op_per_sec":0,"write_op_per_sec":313},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:00.333 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:00.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:00.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 197 20 89404 89384 3.54419 2.02344 0.0595676 0.0440742 2024-09-11T04:59:00.976 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:01.010 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-11T04:58:59.681233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726671552,"num_objects":88710,"num_object_clones":0,"num_object_copies":354840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88737,"num_write_kb":709778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1771,"num_bytes_recovered":14278656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88764,"ondisk_log_size":88764,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2459544,"kb_used_data":1278296,"kb_used_omap":14,"kb_used_meta":1181169,"kb_avail":375027816,"statfs":{"total":386547056640,"available":384028483584,"internally_reserved":0,"allocated":1308975104,"data_stored":1308291864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1209517622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,2,20,7,9,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30826496,"num_objects":3763,"num_object_clones":0,"num_object_copies":15052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":30104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004766"},"pg_stats":[{"pgid":"1.f","version":"31'5496","reported_seq":5538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:57.963775+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:58:57.963775+0000","last_peered":"2024-09-11T04:58:57.963775+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:58:57.963775+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5496,"log_dups_size":0,"ondisk_log_size":5496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45023232,"num_objects":5496,"num_object_clones":0,"num_object_copies":21984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5496,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5608","reported_seq":5670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.765634+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:58:56.765634+0000","last_peered":"2024-09-11T04:58:56.765634+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:58:56.765634+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905056,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":44844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5392","reported_seq":5467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.480305+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:59.480305+0000","last_peered":"2024-09-11T04:58:59.480305+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:59.480305+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44145232,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5465","reported_seq":5528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.798157+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:58:56.798157+0000","last_peered":"2024-09-11T04:58:56.798157+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:58:56.798157+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44733600,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5463,"num_write_kb":43693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5544","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:57.943025+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:58:57.943025+0000","last_peered":"2024-09-11T04:58:57.943025+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:58:57.943025+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45388960,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":44333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5574","reported_seq":5615,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:58.073457+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:58:58.073457+0000","last_peered":"2024-09-11T04:58:58.073457+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:58:58.073457+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45662208,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5574,"num_write_kb":44592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5575","reported_seq":5638,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.810855+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:58:56.810855+0000","last_peered":"2024-09-11T04:58:56.810855+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:58:56.810855+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45618336,"num_objects":5569,"num_object_clones":0,"num_object_copies":22276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":44564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3252}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5602","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.840269+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:58:56.840269+0000","last_peered":"2024-09-11T04:58:56.840269+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:58:56.840269+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45847712,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5622","reported_seq":5678,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:57.940757+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:58:57.940757+0000","last_peered":"2024-09-11T04:58:57.940757+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:58:57.940757+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45984064,"num_objects":5614,"num_object_clones":0,"num_object_copies":22456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5619,"num_write_kb":44929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5575","reported_seq":5652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.349243+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:59.349243+0000","last_peered":"2024-09-11T04:58:59.349243+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:59.349243+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45651104,"num_objects":5573,"num_object_clones":0,"num_object_copies":22292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":44582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3255}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5577","reported_seq":5631,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:58.145575+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:58:58.145575+0000","last_peered":"2024-09-11T04:58:58.145575+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:58:58.145575+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45641456,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5574,"num_write_kb":44580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3249}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5442","reported_seq":8967,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:58.139840+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:58:58.139840+0000","last_peered":"2024-09-11T04:58:58.139840+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:58:58.139840+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44571216,"num_objects":5441,"num_object_clones":0,"num_object_copies":21764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":43527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1745,"num_bytes_recovered":14278656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4949}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5599","reported_seq":5673,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.422918+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:59.422918+0000","last_peered":"2024-09-11T04:58:59.422918+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:59.422918+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45857360,"num_objects":5598,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":44783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5549","reported_seq":5599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:57.823716+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:58:57.823716+0000","last_peered":"2024-09-11T04:58:57.823716+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:58:57.823716+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5549,"log_dups_size":0,"ondisk_log_size":5549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45421728,"num_objects":5545,"num_object_clones":0,"num_object_copies":22180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":44372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3258}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5534","reported_seq":5612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.384061+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:59.384061+0000","last_peered":"2024-09-11T04:58:59.384061+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:59.384061+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45307040,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3213}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5610","reported_seq":5673,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:56.765449+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:58:56.765449+0000","last_peered":"2024-09-11T04:58:56.765449+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:58:56.765449+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45913248,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":44852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3294}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":726671552,"num_objects":88710,"num_object_clones":0,"num_object_copies":354840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88737,"num_write_kb":709778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1771,"num_bytes_recovered":14278656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1309696000,"data_stored":1309696000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88764,"ondisk_log_size":88764,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693512,"kb_used_data":355000,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93678328,"statfs":{"total":96636764160,"available":95926607872,"internally_reserved":0,"allocated":363520000,"data_stored":363349190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":346615504},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,2,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690116,"kb_used_data":356276,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93681724,"statfs":{"total":96636764160,"available":95930085376,"internally_reserved":0,"allocated":364826624,"data_stored":364655814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411860,"kb_used_data":213636,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93959980,"statfs":{"total":96636764160,"available":96215019520,"internally_reserved":0,"allocated":218763264,"data_stored":218592454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":202960594},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664056,"kb_used_data":353384,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93707784,"statfs":{"total":96636764160,"available":95956770816,"internally_reserved":0,"allocated":361865216,"data_stored":361694406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,12,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361963520,"data_stored":361963520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219262976,"data_stored":219262976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364957696,"data_stored":364957696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363511808,"data_stored":363511808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:01.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 198 20 89678 89658 3.5371 2.14062 0.131256 0.0441605 2024-09-11T04:59:02.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 199 20 89969 89949 3.53075 2.27344 0.0766666 0.0442417 2024-09-11T04:59:03.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:59:03.713539+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.044323 2024-09-11T04:59:03.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:59:03.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 200 20 90260 90240 3.52447 2.27344 0.0545478 0.044323 2024-09-11T04:59:04.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:04.712 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 201 20 90525 90505 3.51723 2.07031 0.0649027 0.04441 2024-09-11T04:59:04.791 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:04.792 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:04.812 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-11T04:59:03.682794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":735166656,"num_objects":89747,"num_object_clones":0,"num_object_copies":358988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89774,"num_write_kb":718074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1824,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89801,"ondisk_log_size":89801,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2507948,"kb_used_data":1294760,"kb_used_omap":14,"kb_used_meta":1213105,"kb_avail":374979412,"statfs":{"total":386547056640,"available":383978917888,"internally_reserved":0,"allocated":1325834240,"data_stored":1325151000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1242220086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,7,5,25,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31096832,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3796,"num_write_kb":30368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004974"},"pg_stats":[{"pgid":"1.f","version":"31'5578","reported_seq":5620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.147785+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:03.147785+0000","last_peered":"2024-09-11T04:59:03.147785+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:03.147785+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45694976,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5687","reported_seq":5749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.839274+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:01.839274+0000","last_peered":"2024-09-11T04:59:01.839274+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:01.839274+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46552224,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":45476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5392","reported_seq":5467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.480305+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:59.480305+0000","last_peered":"2024-09-11T04:58:59.480305+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:59.480305+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44145232,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5562","reported_seq":5625,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.797883+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:01.797883+0000","last_peered":"2024-09-11T04:59:01.797883+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:01.797883+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45528224,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5633","reported_seq":5687,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.081399+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:03.081399+0000","last_peered":"2024-09-11T04:59:03.081399+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:03.081399+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5633,"log_dups_size":0,"ondisk_log_size":5633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46118048,"num_objects":5630,"num_object_clones":0,"num_object_copies":22520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5631,"num_write_kb":45045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5657","reported_seq":5698,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.036974+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:03.036974+0000","last_peered":"2024-09-11T04:59:03.036974+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:03.036974+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46342144,"num_objects":5657,"num_object_clones":0,"num_object_copies":22628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5657,"num_write_kb":45256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5657","reported_seq":5720,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.816196+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:01.816196+0000","last_peered":"2024-09-11T04:59:01.816196+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:01.816196+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46290080,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":45220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5718","reported_seq":5780,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.715478+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:01.715478+0000","last_peered":"2024-09-11T04:59:01.715478+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:01.715478+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46797984,"num_objects":5713,"num_object_clones":0,"num_object_copies":22852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5693","reported_seq":5749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.044401+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:03.044401+0000","last_peered":"2024-09-11T04:59:03.044401+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:03.044401+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46565696,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5690,"num_write_kb":45497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5575","reported_seq":5652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.349243+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:59.349243+0000","last_peered":"2024-09-11T04:58:59.349243+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:59.349243+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45651104,"num_objects":5573,"num_object_clones":0,"num_object_copies":22292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":44582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3255}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5658","reported_seq":5714,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.021495+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:03.021495+0000","last_peered":"2024-09-11T04:59:03.021495+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:03.021495+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46305008,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":45228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5532","reported_seq":9163,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.150002+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:03.150002+0000","last_peered":"2024-09-11T04:59:03.150002+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:03.150002+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45308496,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5531,"num_write_kb":44247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1798,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5092}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5599","reported_seq":5673,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.422918+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:59.422918+0000","last_peered":"2024-09-11T04:58:59.422918+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:59.422918+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45857360,"num_objects":5598,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":44783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5639","reported_seq":5684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.088210+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:03.088210+0000","last_peered":"2024-09-11T04:59:03.088210+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:03.088210+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46159008,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":45092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3348}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5534","reported_seq":5612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.384061+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:59.384061+0000","last_peered":"2024-09-11T04:58:59.384061+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:59.384061+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45307040,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3213}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5687","reported_seq":5750,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.834944+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:01.834944+0000","last_peered":"2024-09-11T04:59:01.834944+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:01.834944+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544032,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":45468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":735166656,"num_objects":89747,"num_object_clones":0,"num_object_copies":358988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89774,"num_write_kb":718074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1824,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1326534656,"data_stored":1326534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89801,"ondisk_log_size":89801,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703180,"kb_used_data":360504,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93668660,"statfs":{"total":96636764160,"available":95916707840,"internally_reserved":0,"allocated":369156096,"data_stored":368985286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,14,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690116,"kb_used_data":356276,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93681724,"statfs":{"total":96636764160,"available":95930085376,"internally_reserved":0,"allocated":364826624,"data_stored":364655814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421624,"kb_used_data":219240,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93950216,"statfs":{"total":96636764160,"available":96205021184,"internally_reserved":0,"allocated":224501760,"data_stored":224330950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207220434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693028,"kb_used_data":358740,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93678812,"statfs":{"total":96636764160,"available":95927103488,"internally_reserved":0,"allocated":367349760,"data_stored":367178950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342290127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367517696,"data_stored":367517696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225009664,"data_stored":225009664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364957696,"data_stored":364957696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":369049600,"data_stored":369049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:04.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:05.486 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:05.513 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-11T04:59:03.682794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":735166656,"num_objects":89747,"num_object_clones":0,"num_object_copies":358988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89774,"num_write_kb":718074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1824,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89801,"ondisk_log_size":89801,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2507948,"kb_used_data":1294760,"kb_used_omap":14,"kb_used_meta":1213105,"kb_avail":374979412,"statfs":{"total":386547056640,"available":383978917888,"internally_reserved":0,"allocated":1325834240,"data_stored":1325151000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1242220086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,7,5,25,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31096832,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3796,"num_write_kb":30368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004974"},"pg_stats":[{"pgid":"1.f","version":"31'5578","reported_seq":5620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.147785+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:03.147785+0000","last_peered":"2024-09-11T04:59:03.147785+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:03.147785+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45694976,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5687","reported_seq":5749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.839274+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:01.839274+0000","last_peered":"2024-09-11T04:59:01.839274+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:01.839274+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46552224,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":45476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5392","reported_seq":5467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.480305+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:58:59.480305+0000","last_peered":"2024-09-11T04:58:59.480305+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:58:59.480305+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44145232,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5562","reported_seq":5625,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.797883+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:01.797883+0000","last_peered":"2024-09-11T04:59:01.797883+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:01.797883+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45528224,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5633","reported_seq":5687,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.081399+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:03.081399+0000","last_peered":"2024-09-11T04:59:03.081399+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:03.081399+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5633,"log_dups_size":0,"ondisk_log_size":5633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46118048,"num_objects":5630,"num_object_clones":0,"num_object_copies":22520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5631,"num_write_kb":45045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5657","reported_seq":5698,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.036974+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:03.036974+0000","last_peered":"2024-09-11T04:59:03.036974+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:03.036974+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46342144,"num_objects":5657,"num_object_clones":0,"num_object_copies":22628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5657,"num_write_kb":45256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5657","reported_seq":5720,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.816196+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:01.816196+0000","last_peered":"2024-09-11T04:59:01.816196+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:01.816196+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46290080,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":45220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5718","reported_seq":5780,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.715478+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:01.715478+0000","last_peered":"2024-09-11T04:59:01.715478+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:01.715478+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46797984,"num_objects":5713,"num_object_clones":0,"num_object_copies":22852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5693","reported_seq":5749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.044401+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:03.044401+0000","last_peered":"2024-09-11T04:59:03.044401+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:03.044401+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46565696,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5690,"num_write_kb":45497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5575","reported_seq":5652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.349243+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:58:59.349243+0000","last_peered":"2024-09-11T04:58:59.349243+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:58:59.349243+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45651104,"num_objects":5573,"num_object_clones":0,"num_object_copies":22292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":44582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3255}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5658","reported_seq":5714,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.021495+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:03.021495+0000","last_peered":"2024-09-11T04:59:03.021495+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:03.021495+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46305008,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":45228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5532","reported_seq":9163,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.150002+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:03.150002+0000","last_peered":"2024-09-11T04:59:03.150002+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:03.150002+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45308496,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5531,"num_write_kb":44247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1798,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5092}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5599","reported_seq":5673,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.422918+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:58:59.422918+0000","last_peered":"2024-09-11T04:58:59.422918+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:58:59.422918+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45857360,"num_objects":5598,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":44783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5639","reported_seq":5684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.088210+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:03.088210+0000","last_peered":"2024-09-11T04:59:03.088210+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:03.088210+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46159008,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":45092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3348}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5534","reported_seq":5612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:58:59.384061+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:58:59.384061+0000","last_peered":"2024-09-11T04:58:59.384061+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:58:59.384061+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45307040,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3213}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5687","reported_seq":5750,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.834944+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:01.834944+0000","last_peered":"2024-09-11T04:59:01.834944+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:01.834944+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544032,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":45468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":735166656,"num_objects":89747,"num_object_clones":0,"num_object_copies":358988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89774,"num_write_kb":718074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1824,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1326534656,"data_stored":1326534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89801,"ondisk_log_size":89801,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703180,"kb_used_data":360504,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93668660,"statfs":{"total":96636764160,"available":95916707840,"internally_reserved":0,"allocated":369156096,"data_stored":368985286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,14,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690116,"kb_used_data":356276,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93681724,"statfs":{"total":96636764160,"available":95930085376,"internally_reserved":0,"allocated":364826624,"data_stored":364655814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421624,"kb_used_data":219240,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93950216,"statfs":{"total":96636764160,"available":96205021184,"internally_reserved":0,"allocated":224501760,"data_stored":224330950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207220434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693028,"kb_used_data":358740,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93678812,"statfs":{"total":96636764160,"available":95927103488,"internally_reserved":0,"allocated":367349760,"data_stored":367178950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342290127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367517696,"data_stored":367517696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225009664,"data_stored":225009664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364957696,"data_stored":364957696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":369049600,"data_stored":369049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:05.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:05.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5578", 'reported_seq': 5620, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:03.147785+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:03.147785+0000', 'last_peered': '2024-09-11T04:59:03.147785+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:03.147785+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5578, 'log_dups_size': 0, 'ondisk_log_size': 5578, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45694976, 'num_objects': 5578, 'num_object_clones': 0, 'num_object_copies': 22312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5578, 'num_write_kb': 44624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3266}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:05.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:05.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5687", 'reported_seq': 5749, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:01.839274+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:01.839274+0000', 'last_peered': '2024-09-11T04:59:01.839274+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:01.839274+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5687, 'log_dups_size': 0, 'ondisk_log_size': 5687, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46552224, 'num_objects': 5683, 'num_object_clones': 0, 'num_object_copies': 22732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5685, 'num_write_kb': 45476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:05.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:05.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5392", 'reported_seq': 5467, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:59.480305+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:58:59.480305+0000', 'last_peered': '2024-09-11T04:58:59.480305+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:58:59.480305+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5392, 'log_dups_size': 0, 'ondisk_log_size': 5392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44145232, 'num_objects': 5389, 'num_object_clones': 0, 'num_object_copies': 21556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5391, 'num_write_kb': 43118, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3169}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:05.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:05.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5562", 'reported_seq': 5625, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:01.797883+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:01.797883+0000', 'last_peered': '2024-09-11T04:59:01.797883+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:01.797883+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5562, 'log_dups_size': 0, 'ondisk_log_size': 5562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45528224, 'num_objects': 5558, 'num_object_clones': 0, 'num_object_copies': 22232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5560, 'num_write_kb': 44469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5633", 'reported_seq': 5687, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:03.081399+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:03.081399+0000', 'last_peered': '2024-09-11T04:59:03.081399+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:03.081399+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5633, 'log_dups_size': 0, 'ondisk_log_size': 5633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46118048, 'num_objects': 5630, 'num_object_clones': 0, 'num_object_copies': 22520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5631, 'num_write_kb': 45045, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3380}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5657", 'reported_seq': 5698, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:03.036974+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:03.036974+0000', 'last_peered': '2024-09-11T04:59:03.036974+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:03.036974+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5657, 'log_dups_size': 0, 'ondisk_log_size': 5657, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46342144, 'num_objects': 5657, 'num_object_clones': 0, 'num_object_copies': 22628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5657, 'num_write_kb': 45256, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3336}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5657", 'reported_seq': 5720, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:01.816196+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:01.816196+0000', 'last_peered': '2024-09-11T04:59:01.816196+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:01.816196+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5657, 'log_dups_size': 0, 'ondisk_log_size': 5657, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46290080, 'num_objects': 5651, 'num_object_clones': 0, 'num_object_copies': 22604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5655, 'num_write_kb': 45220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3334}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5718", 'reported_seq': 5780, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:01.715478+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:59:01.715478+0000', 'last_peered': '2024-09-11T04:59:01.715478+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:01.715478+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5718, 'log_dups_size': 0, 'ondisk_log_size': 5718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46797984, 'num_objects': 5713, 'num_object_clones': 0, 'num_object_copies': 22852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5716, 'num_write_kb': 45716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5693", 'reported_seq': 5749, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:03.044401+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:03.044401+0000', 'last_peered': '2024-09-11T04:59:03.044401+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:03.044401+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5693, 'log_dups_size': 0, 'ondisk_log_size': 5693, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46565696, 'num_objects': 5685, 'num_object_clones': 0, 'num_object_copies': 22740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5690, 'num_write_kb': 45497, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:05.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5575", 'reported_seq': 5652, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:59.349243+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:58:59.349243+0000', 'last_peered': '2024-09-11T04:58:59.349243+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:58:59.349243+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5575, 'log_dups_size': 0, 'ondisk_log_size': 5575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45651104, 'num_objects': 5573, 'num_object_clones': 0, 'num_object_copies': 22292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5573, 'num_write_kb': 44582, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3255}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5658", 'reported_seq': 5714, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:03.021495+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:03.021495+0000', 'last_peered': '2024-09-11T04:59:03.021495+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:03.021495+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5658, 'log_dups_size': 0, 'ondisk_log_size': 5658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46305008, 'num_objects': 5653, 'num_object_clones': 0, 'num_object_copies': 22612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5655, 'num_write_kb': 45228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3330}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5532", 'reported_seq': 9163, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:03.150002+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:59:03.150002+0000', 'last_peered': '2024-09-11T04:59:03.150002+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:59:03.150002+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5532, 'log_dups_size': 0, 'ondisk_log_size': 5532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45308496, 'num_objects': 5531, 'num_object_clones': 0, 'num_object_copies': 22124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 439, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5531, 'num_write_kb': 44247, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1798, 'num_bytes_recovered': 14712832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5092}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5599", 'reported_seq': 5673, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:59.422918+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:58:59.422918+0000', 'last_peered': '2024-09-11T04:58:59.422918+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:58:59.422918+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5599, 'log_dups_size': 0, 'ondisk_log_size': 5599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45857360, 'num_objects': 5598, 'num_object_clones': 0, 'num_object_copies': 22392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5598, 'num_write_kb': 44783, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5639", 'reported_seq': 5684, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:03.088210+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:03.088210+0000', 'last_peered': '2024-09-11T04:59:03.088210+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:03.088210+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5639, 'log_dups_size': 0, 'ondisk_log_size': 5639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46159008, 'num_objects': 5635, 'num_object_clones': 0, 'num_object_copies': 22540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5637, 'num_write_kb': 45092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5534", 'reported_seq': 5612, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:58:59.384061+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:58:59.384061+0000', 'last_peered': '2024-09-11T04:58:59.384061+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:58:59.384061+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5534, 'log_dups_size': 0, 'ondisk_log_size': 5534, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45307040, 'num_objects': 5531, 'num_object_clones': 0, 'num_object_copies': 22124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5532, 'num_write_kb': 44253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3213}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:05.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:05.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5687", 'reported_seq': 5750, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:01.834944+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:01.834944+0000', 'last_peered': '2024-09-11T04:59:01.834944+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:01.834944+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5687, 'log_dups_size': 0, 'ondisk_log_size': 5687, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46544032, 'num_objects': 5682, 'num_object_clones': 0, 'num_object_copies': 22728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5685, 'num_write_kb': 45468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3371}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:05.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:05.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 202 20 90778 90758 3.5096 1.97656 0.0697694 0.0445074 2024-09-11T04:59:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:06.273 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35086/358988 objects degraded (9.774%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":273,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89747,"data_bytes":735166656,"bytes_used":2568138752,"bytes_avail":383978917888,"bytes_total":386547056640,"degraded_objects":35086,"degraded_total":358988,"degraded_ratio":0.097735857465987716,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":141253,"recovering_keys_per_sec":0,"num_objects_recovered":207,"num_bytes_recovered":1695744,"num_keys_recovered":0,"write_bytes_sec":2590328,"read_op_per_sec":0,"write_op_per_sec":316},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:06.301 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:06.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:06.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 203 20 91031 91011 3.50205 1.97656 0.0645516 0.0446015 2024-09-11T04:59:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:06.989 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-11T04:59:05.683826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738099392,"num_objects":90105,"num_object_clones":0,"num_object_copies":360420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90132,"num_write_kb":720938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1824,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90159,"ondisk_log_size":90159,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2517564,"kb_used_data":1300216,"kb_used_omap":14,"kb_used_meta":1217265,"kb_avail":374969796,"statfs":{"total":386547056640,"available":383969071104,"internally_reserved":0,"allocated":1331421184,"data_stored":1330737944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1246479926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,3,6,24,5,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25681920,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3135,"num_write_kb":25080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005009"},"pg_stats":[{"pgid":"1.f","version":"31'5578","reported_seq":5620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.147785+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:03.147785+0000","last_peered":"2024-09-11T04:59:03.147785+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:03.147785+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45694976,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5687","reported_seq":5749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.839274+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:01.839274+0000","last_peered":"2024-09-11T04:59:01.839274+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:01.839274+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46552224,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":45476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5476","reported_seq":5550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:04.474163+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:04.474163+0000","last_peered":"2024-09-11T04:59:04.474163+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:04.474163+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44833360,"num_objects":5473,"num_object_clones":0,"num_object_copies":21892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":43790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3253}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5562","reported_seq":5625,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.797883+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:01.797883+0000","last_peered":"2024-09-11T04:59:01.797883+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:01.797883+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45528224,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5633","reported_seq":5687,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.081399+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:03.081399+0000","last_peered":"2024-09-11T04:59:03.081399+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:03.081399+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5633,"log_dups_size":0,"ondisk_log_size":5633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46118048,"num_objects":5630,"num_object_clones":0,"num_object_copies":22520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5631,"num_write_kb":45045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5657","reported_seq":5698,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.036974+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:03.036974+0000","last_peered":"2024-09-11T04:59:03.036974+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:03.036974+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46342144,"num_objects":5657,"num_object_clones":0,"num_object_copies":22628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5657,"num_write_kb":45256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5657","reported_seq":5720,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.816196+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:01.816196+0000","last_peered":"2024-09-11T04:59:01.816196+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:01.816196+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46290080,"num_objects":5651,"num_object_clones":0,"num_object_copies":22604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":45220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5718","reported_seq":5780,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.715478+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:01.715478+0000","last_peered":"2024-09-11T04:59:01.715478+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:01.715478+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46797984,"num_objects":5713,"num_object_clones":0,"num_object_copies":22852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5693","reported_seq":5749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.044401+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:03.044401+0000","last_peered":"2024-09-11T04:59:03.044401+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:03.044401+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46565696,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5690,"num_write_kb":45497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5672","reported_seq":5749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:04.529372+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:04.529372+0000","last_peered":"2024-09-11T04:59:04.529372+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:04.529372+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46445728,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5670,"num_write_kb":45358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5658","reported_seq":5714,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.021495+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:03.021495+0000","last_peered":"2024-09-11T04:59:03.021495+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:03.021495+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46305008,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":45228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5532","reported_seq":9163,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.150002+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:03.150002+0000","last_peered":"2024-09-11T04:59:03.150002+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:03.150002+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45308496,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5531,"num_write_kb":44247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1798,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5092}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5687","reported_seq":5759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:04.229892+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:04.229892+0000","last_peered":"2024-09-11T04:59:04.229892+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:04.229892+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46578256,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":45487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5639","reported_seq":5684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:03.088210+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:03.088210+0000","last_peered":"2024-09-11T04:59:03.088210+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:03.088210+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46159008,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":45092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3348}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5623","reported_seq":5700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:04.300370+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:04.300370+0000","last_peered":"2024-09-11T04:59:04.300370+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:04.300370+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46036128,"num_objects":5620,"num_object_clones":0,"num_object_copies":22480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5621,"num_write_kb":44965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5687","reported_seq":5750,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:01.834944+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:01.834944+0000","last_peered":"2024-09-11T04:59:01.834944+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:01.834944+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544032,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":45468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738099392,"num_objects":90105,"num_object_clones":0,"num_object_copies":360420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90132,"num_write_kb":720938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1824,"num_bytes_recovered":14712832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1332195328,"data_stored":1332195328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90159,"ondisk_log_size":90159,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703180,"kb_used_data":360504,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93668660,"statfs":{"total":96636764160,"available":95916707840,"internally_reserved":0,"allocated":369156096,"data_stored":368985286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,14,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699732,"kb_used_data":361732,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93672108,"statfs":{"total":96636764160,"available":95920238592,"internally_reserved":0,"allocated":370413568,"data_stored":370242758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421624,"kb_used_data":219240,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93950216,"statfs":{"total":96636764160,"available":96205021184,"internally_reserved":0,"allocated":224501760,"data_stored":224330950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":207220434},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693028,"kb_used_data":358740,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93678812,"statfs":{"total":96636764160,"available":95927103488,"internally_reserved":0,"allocated":367349760,"data_stored":367178950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342290127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367517696,"data_stored":367517696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225009664,"data_stored":225009664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370618368,"data_stored":370618368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":369049600,"data_stored":369049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:07.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 204 20 91309 91289 3.49553 2.17188 0.0642367 0.0446877 2024-09-11T04:59:08.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 205 20 91590 91570 3.48919 2.19531 0.0344308 0.0447665 2024-09-11T04:59:09.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 206 20 91874 91854 3.48302 2.21875 0.0651057 0.0448463 2024-09-11T04:59:09.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:10.601 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:10.601 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:10.615 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-11T04:59:09.685439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":749240512,"num_objects":91465,"num_object_clones":0,"num_object_copies":365860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91492,"num_write_kb":731818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1885,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91519,"ondisk_log_size":91519,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2557352,"kb_used_data":1321960,"kb_used_omap":14,"kb_used_meta":1235313,"kb_avail":374930008,"statfs":{"total":386547056640,"available":383928328192,"internally_reserved":0,"allocated":1353687040,"data_stored":1353003800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1264961078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,0,4,8,7,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30564352,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3731,"num_write_kb":29848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004937"},"pg_stats":[{"pgid":"1.f","version":"31'5663","reported_seq":5703,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.125056+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:08.125056+0000","last_peered":"2024-09-11T04:59:08.125056+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:08.125056+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46391296,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":45304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5769","reported_seq":5831,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.894568+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:06.894568+0000","last_peered":"2024-09-11T04:59:06.894568+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:06.894568+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47223968,"num_objects":5765,"num_object_clones":0,"num_object_copies":23060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5568","reported_seq":5643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.460920+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:09.460920+0000","last_peered":"2024-09-11T04:59:09.460920+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:09.460920+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45587024,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5567,"num_write_kb":44526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3345}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5622","reported_seq":5685,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.782339+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:06.782339+0000","last_peered":"2024-09-11T04:59:06.782339+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:06.782339+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46019744,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5620,"num_write_kb":44949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5713","reported_seq":5767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.086717+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:08.086717+0000","last_peered":"2024-09-11T04:59:08.086717+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:08.086717+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5713,"log_dups_size":0,"ondisk_log_size":5713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46773408,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5711,"num_write_kb":45685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5738","reported_seq":5779,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:07.926027+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:07.926027+0000","last_peered":"2024-09-11T04:59:07.926027+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:07.926027+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47005696,"num_objects":5738,"num_object_clones":0,"num_object_copies":22952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":45904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5741","reported_seq":5804,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.775775+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:06.775775+0000","last_peered":"2024-09-11T04:59:06.775775+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:06.775775+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5741,"log_dups_size":0,"ondisk_log_size":5741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46978208,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5739,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3418}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5807","reported_seq":5869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.863108+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:06.863108+0000","last_peered":"2024-09-11T04:59:06.863108+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:06.863108+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5807,"log_dups_size":0,"ondisk_log_size":5807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47527072,"num_objects":5802,"num_object_clones":0,"num_object_copies":23208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5805,"num_write_kb":46428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5798","reported_seq":5854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.073800+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:08.073800+0000","last_peered":"2024-09-11T04:59:08.073800+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:08.073800+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5798,"log_dups_size":0,"ondisk_log_size":5798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47425856,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":46337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5770","reported_seq":5848,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.593333+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:09.593333+0000","last_peered":"2024-09-11T04:59:09.593333+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:09.593333+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47248544,"num_objects":5768,"num_object_clones":0,"num_object_copies":23072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3450}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5722","reported_seq":5777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.146280+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:08.146280+0000","last_peered":"2024-09-11T04:59:08.146280+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:08.146280+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46829296,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5628","reported_seq":9379,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.150724+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:08.150724+0000","last_peered":"2024-09-11T04:59:08.150724+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:08.150724+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46094928,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":45015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1859,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5249}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5775","reported_seq":5849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.569153+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:09.569153+0000","last_peered":"2024-09-11T04:59:09.569153+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:09.569153+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47299152,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":46191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3442}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5720","reported_seq":5768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.090647+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:08.090647+0000","last_peered":"2024-09-11T04:59:08.090647+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:08.090647+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5720,"log_dups_size":0,"ondisk_log_size":5720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822560,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5718,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5708","reported_seq":5786,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.446491+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:09.446491+0000","last_peered":"2024-09-11T04:59:09.446491+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:09.446491+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732448,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3387}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5777","reported_seq":5840,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.840187+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:06.840187+0000","last_peered":"2024-09-11T04:59:06.840187+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:06.840187+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281312,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":46188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":749240512,"num_objects":91465,"num_object_clones":0,"num_object_copies":365860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91492,"num_write_kb":731818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1885,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1354637312,"data_stored":1354637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91519,"ondisk_log_size":91519,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712704,"kb_used_data":365872,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93659136,"statfs":{"total":96636764160,"available":95906955264,"internally_reserved":0,"allocated":374652928,"data_stored":374482118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,2,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709284,"kb_used_data":367124,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93662556,"statfs":{"total":96636764160,"available":95910457344,"internally_reserved":0,"allocated":375934976,"data_stored":375764166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,0,1,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432764,"kb_used_data":224812,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93939076,"statfs":{"total":96636764160,"available":96193613824,"internally_reserved":0,"allocated":230207488,"data_stored":230036678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":212922066},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702600,"kb_used_data":364152,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93669240,"statfs":{"total":96636764160,"available":95917301760,"internally_reserved":0,"allocated":372891648,"data_stored":372720838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373059584,"data_stored":373059584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230846464,"data_stored":230846464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376193024,"data_stored":376193024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374538240,"data_stored":374538240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:10.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:10.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 207 20 92124 92104 3.47563 1.95312 0.127248 0.0449403 2024-09-11T04:59:11.251 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:11.252 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:11.279 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-11T04:59:09.685439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":749240512,"num_objects":91465,"num_object_clones":0,"num_object_copies":365860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91492,"num_write_kb":731818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1885,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91519,"ondisk_log_size":91519,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2557352,"kb_used_data":1321960,"kb_used_omap":14,"kb_used_meta":1235313,"kb_avail":374930008,"statfs":{"total":386547056640,"available":383928328192,"internally_reserved":0,"allocated":1353687040,"data_stored":1353003800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1264961078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,0,4,8,7,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30564352,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3731,"num_write_kb":29848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004937"},"pg_stats":[{"pgid":"1.f","version":"31'5663","reported_seq":5703,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.125056+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:08.125056+0000","last_peered":"2024-09-11T04:59:08.125056+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:08.125056+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46391296,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":45304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5769","reported_seq":5831,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.894568+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:06.894568+0000","last_peered":"2024-09-11T04:59:06.894568+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:06.894568+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47223968,"num_objects":5765,"num_object_clones":0,"num_object_copies":23060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5568","reported_seq":5643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.460920+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:09.460920+0000","last_peered":"2024-09-11T04:59:09.460920+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:09.460920+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45587024,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5567,"num_write_kb":44526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3345}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5622","reported_seq":5685,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.782339+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:06.782339+0000","last_peered":"2024-09-11T04:59:06.782339+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:06.782339+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46019744,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5620,"num_write_kb":44949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5713","reported_seq":5767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.086717+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:08.086717+0000","last_peered":"2024-09-11T04:59:08.086717+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:08.086717+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5713,"log_dups_size":0,"ondisk_log_size":5713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46773408,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5711,"num_write_kb":45685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5738","reported_seq":5779,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:07.926027+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:07.926027+0000","last_peered":"2024-09-11T04:59:07.926027+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:07.926027+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47005696,"num_objects":5738,"num_object_clones":0,"num_object_copies":22952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":45904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5741","reported_seq":5804,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.775775+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:06.775775+0000","last_peered":"2024-09-11T04:59:06.775775+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:06.775775+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5741,"log_dups_size":0,"ondisk_log_size":5741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46978208,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5739,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3418}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5807","reported_seq":5869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.863108+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:06.863108+0000","last_peered":"2024-09-11T04:59:06.863108+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:06.863108+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5807,"log_dups_size":0,"ondisk_log_size":5807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47527072,"num_objects":5802,"num_object_clones":0,"num_object_copies":23208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5805,"num_write_kb":46428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5798","reported_seq":5854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.073800+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:08.073800+0000","last_peered":"2024-09-11T04:59:08.073800+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:08.073800+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5798,"log_dups_size":0,"ondisk_log_size":5798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47425856,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":46337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5770","reported_seq":5848,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.593333+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:09.593333+0000","last_peered":"2024-09-11T04:59:09.593333+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:09.593333+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47248544,"num_objects":5768,"num_object_clones":0,"num_object_copies":23072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3450}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5722","reported_seq":5777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.146280+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:08.146280+0000","last_peered":"2024-09-11T04:59:08.146280+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:08.146280+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46829296,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5628","reported_seq":9379,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.150724+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:08.150724+0000","last_peered":"2024-09-11T04:59:08.150724+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:08.150724+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46094928,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":45015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1859,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5249}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5775","reported_seq":5849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.569153+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:09.569153+0000","last_peered":"2024-09-11T04:59:09.569153+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:09.569153+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47299152,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":46191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3442}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5720","reported_seq":5768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.090647+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:08.090647+0000","last_peered":"2024-09-11T04:59:08.090647+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:08.090647+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5720,"log_dups_size":0,"ondisk_log_size":5720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822560,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5718,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5708","reported_seq":5786,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.446491+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:09.446491+0000","last_peered":"2024-09-11T04:59:09.446491+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:09.446491+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732448,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3387}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5777","reported_seq":5840,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.840187+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:06.840187+0000","last_peered":"2024-09-11T04:59:06.840187+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:06.840187+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281312,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":46188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":749240512,"num_objects":91465,"num_object_clones":0,"num_object_copies":365860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91492,"num_write_kb":731818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1885,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1354637312,"data_stored":1354637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91519,"ondisk_log_size":91519,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712704,"kb_used_data":365872,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93659136,"statfs":{"total":96636764160,"available":95906955264,"internally_reserved":0,"allocated":374652928,"data_stored":374482118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,2,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709284,"kb_used_data":367124,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93662556,"statfs":{"total":96636764160,"available":95910457344,"internally_reserved":0,"allocated":375934976,"data_stored":375764166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,0,1,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432764,"kb_used_data":224812,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93939076,"statfs":{"total":96636764160,"available":96193613824,"internally_reserved":0,"allocated":230207488,"data_stored":230036678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":212922066},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702600,"kb_used_data":364152,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93669240,"statfs":{"total":96636764160,"available":95917301760,"internally_reserved":0,"allocated":372891648,"data_stored":372720838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373059584,"data_stored":373059584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230846464,"data_stored":230846464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376193024,"data_stored":376193024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374538240,"data_stored":374538240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5663", 'reported_seq': 5703, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:08.125056+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:08.125056+0000', 'last_peered': '2024-09-11T04:59:08.125056+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:08.125056+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5663, 'log_dups_size': 0, 'ondisk_log_size': 5663, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46391296, 'num_objects': 5663, 'num_object_clones': 0, 'num_object_copies': 22652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5663, 'num_write_kb': 45304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5769", 'reported_seq': 5831, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:06.894568+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:06.894568+0000', 'last_peered': '2024-09-11T04:59:06.894568+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:06.894568+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5769, 'log_dups_size': 0, 'ondisk_log_size': 5769, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47223968, 'num_objects': 5765, 'num_object_clones': 0, 'num_object_copies': 23060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5767, 'num_write_kb': 46132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3486}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5568", 'reported_seq': 5643, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:09.460920+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:09.460920+0000', 'last_peered': '2024-09-11T04:59:09.460920+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:09.460920+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5568, 'log_dups_size': 0, 'ondisk_log_size': 5568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45587024, 'num_objects': 5565, 'num_object_clones': 0, 'num_object_copies': 22260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5567, 'num_write_kb': 44526, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3345}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5622", 'reported_seq': 5685, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:06.782339+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:06.782339+0000', 'last_peered': '2024-09-11T04:59:06.782339+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:06.782339+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5622, 'log_dups_size': 0, 'ondisk_log_size': 5622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46019744, 'num_objects': 5618, 'num_object_clones': 0, 'num_object_copies': 22472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5620, 'num_write_kb': 44949, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5713", 'reported_seq': 5767, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:08.086717+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:08.086717+0000', 'last_peered': '2024-09-11T04:59:08.086717+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:08.086717+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5713, 'log_dups_size': 0, 'ondisk_log_size': 5713, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46773408, 'num_objects': 5710, 'num_object_clones': 0, 'num_object_copies': 22840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5711, 'num_write_kb': 45685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3460}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5738", 'reported_seq': 5779, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:07.926027+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:07.926027+0000', 'last_peered': '2024-09-11T04:59:07.926027+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:07.926027+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5738, 'log_dups_size': 0, 'ondisk_log_size': 5738, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47005696, 'num_objects': 5738, 'num_object_clones': 0, 'num_object_copies': 22952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5738, 'num_write_kb': 45904, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3417}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5741", 'reported_seq': 5804, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:06.775775+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:06.775775+0000', 'last_peered': '2024-09-11T04:59:06.775775+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:06.775775+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5741, 'log_dups_size': 0, 'ondisk_log_size': 5741, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46978208, 'num_objects': 5735, 'num_object_clones': 0, 'num_object_copies': 22940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5739, 'num_write_kb': 45892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3418}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5807", 'reported_seq': 5869, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:06.863108+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:59:06.863108+0000', 'last_peered': '2024-09-11T04:59:06.863108+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:06.863108+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5807, 'log_dups_size': 0, 'ondisk_log_size': 5807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47527072, 'num_objects': 5802, 'num_object_clones': 0, 'num_object_copies': 23208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5805, 'num_write_kb': 46428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3441}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5798", 'reported_seq': 5854, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:08.073800+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:08.073800+0000', 'last_peered': '2024-09-11T04:59:08.073800+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:08.073800+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5798, 'log_dups_size': 0, 'ondisk_log_size': 5798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47425856, 'num_objects': 5790, 'num_object_clones': 0, 'num_object_copies': 23160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5795, 'num_write_kb': 46337, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3396}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5770", 'reported_seq': 5848, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:09.593333+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:09.593333+0000', 'last_peered': '2024-09-11T04:59:09.593333+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:09.593333+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5770, 'log_dups_size': 0, 'ondisk_log_size': 5770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47248544, 'num_objects': 5768, 'num_object_clones': 0, 'num_object_copies': 23072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5768, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5768, 'num_write_kb': 46142, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3450}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5722", 'reported_seq': 5777, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:08.146280+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:08.146280+0000', 'last_peered': '2024-09-11T04:59:08.146280+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:08.146280+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5722, 'log_dups_size': 0, 'ondisk_log_size': 5722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46829296, 'num_objects': 5717, 'num_object_clones': 0, 'num_object_copies': 22868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5719, 'num_write_kb': 45740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5628", 'reported_seq': 9379, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:08.150724+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:59:08.150724+0000', 'last_peered': '2024-09-11T04:59:08.150724+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:59:08.150724+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5628, 'log_dups_size': 0, 'ondisk_log_size': 5628, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46094928, 'num_objects': 5627, 'num_object_clones': 0, 'num_object_copies': 22508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 378, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5627, 'num_write_kb': 45015, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1859, 'num_bytes_recovered': 15212544, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5249}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5775", 'reported_seq': 5849, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:09.569153+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:09.569153+0000', 'last_peered': '2024-09-11T04:59:09.569153+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:09.569153+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5775, 'log_dups_size': 0, 'ondisk_log_size': 5775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47299152, 'num_objects': 5774, 'num_object_clones': 0, 'num_object_copies': 23096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5774, 'num_write_kb': 46191, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3442}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5720", 'reported_seq': 5768, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:08.090647+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:08.090647+0000', 'last_peered': '2024-09-11T04:59:08.090647+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:08.090647+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5720, 'log_dups_size': 0, 'ondisk_log_size': 5720, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46822560, 'num_objects': 5716, 'num_object_clones': 0, 'num_object_copies': 22864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5718, 'num_write_kb': 45740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5708", 'reported_seq': 5786, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:09.446491+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:09.446491+0000', 'last_peered': '2024-09-11T04:59:09.446491+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:09.446491+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5708, 'log_dups_size': 0, 'ondisk_log_size': 5708, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46732448, 'num_objects': 5705, 'num_object_clones': 0, 'num_object_copies': 22820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5706, 'num_write_kb': 45645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3387}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5777", 'reported_seq': 5840, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:06.840187+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:06.840187+0000', 'last_peered': '2024-09-11T04:59:06.840187+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:06.840187+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5777, 'log_dups_size': 0, 'ondisk_log_size': 5777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47281312, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 23088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5775, 'num_write_kb': 46188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3461}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:11.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:11.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 208 20 92374 92354 3.46831 1.95312 0.0820891 0.0450335 2024-09-11T04:59:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:12.078 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35025/365860 objects degraded (9.573%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":279,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91465,"data_bytes":749240512,"bytes_used":2618728448,"bytes_avail":383928328192,"bytes_total":386547056640,"degraded_objects":35025,"degraded_total":365860,"degraded_ratio":0.095733340622095889,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":77792,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"write_bytes_sec":1879974,"read_op_per_sec":0,"write_op_per_sec":229},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:12.102 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:12.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:12.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 209 20 92619 92599 3.46087 1.91406 0.0521824 0.0451322 2024-09-11T04:59:12.752 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:12.754 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:12.780 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-11T04:59:11.686164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":749240512,"num_objects":91465,"num_object_clones":0,"num_object_copies":365860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91492,"num_write_kb":731818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1885,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91519,"ondisk_log_size":91519,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2557352,"kb_used_data":1321960,"kb_used_omap":14,"kb_used_meta":1235313,"kb_avail":374930008,"statfs":{"total":386547056640,"available":383928328192,"internally_reserved":0,"allocated":1353687040,"data_stored":1353003800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1264961078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,0,4,8,7,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22568960,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2755,"num_write_kb":22040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004931"},"pg_stats":[{"pgid":"1.f","version":"31'5663","reported_seq":5703,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.125056+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:08.125056+0000","last_peered":"2024-09-11T04:59:08.125056+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:08.125056+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46391296,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":45304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5769","reported_seq":5831,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.894568+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:06.894568+0000","last_peered":"2024-09-11T04:59:06.894568+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:06.894568+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47223968,"num_objects":5765,"num_object_clones":0,"num_object_copies":23060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5568","reported_seq":5643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.460920+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:09.460920+0000","last_peered":"2024-09-11T04:59:09.460920+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:09.460920+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45587024,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5567,"num_write_kb":44526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3345}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5622","reported_seq":5685,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.782339+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:06.782339+0000","last_peered":"2024-09-11T04:59:06.782339+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:06.782339+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46019744,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5620,"num_write_kb":44949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5713","reported_seq":5767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.086717+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:08.086717+0000","last_peered":"2024-09-11T04:59:08.086717+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:08.086717+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5713,"log_dups_size":0,"ondisk_log_size":5713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46773408,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5711,"num_write_kb":45685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5738","reported_seq":5779,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:07.926027+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:07.926027+0000","last_peered":"2024-09-11T04:59:07.926027+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:07.926027+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47005696,"num_objects":5738,"num_object_clones":0,"num_object_copies":22952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":45904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5741","reported_seq":5804,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.775775+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:06.775775+0000","last_peered":"2024-09-11T04:59:06.775775+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:06.775775+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5741,"log_dups_size":0,"ondisk_log_size":5741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46978208,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5739,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3418}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5807","reported_seq":5869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.863108+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:06.863108+0000","last_peered":"2024-09-11T04:59:06.863108+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:06.863108+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5807,"log_dups_size":0,"ondisk_log_size":5807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47527072,"num_objects":5802,"num_object_clones":0,"num_object_copies":23208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5805,"num_write_kb":46428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5798","reported_seq":5854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.073800+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:08.073800+0000","last_peered":"2024-09-11T04:59:08.073800+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:08.073800+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5798,"log_dups_size":0,"ondisk_log_size":5798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47425856,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":46337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5770","reported_seq":5848,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.593333+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:09.593333+0000","last_peered":"2024-09-11T04:59:09.593333+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:09.593333+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47248544,"num_objects":5768,"num_object_clones":0,"num_object_copies":23072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3450}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5722","reported_seq":5777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.146280+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:08.146280+0000","last_peered":"2024-09-11T04:59:08.146280+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:08.146280+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46829296,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5628","reported_seq":9379,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.150724+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:08.150724+0000","last_peered":"2024-09-11T04:59:08.150724+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:08.150724+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46094928,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":45015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1859,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5249}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5775","reported_seq":5849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.569153+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:09.569153+0000","last_peered":"2024-09-11T04:59:09.569153+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:09.569153+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47299152,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":46191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3442}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5720","reported_seq":5768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:08.090647+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:08.090647+0000","last_peered":"2024-09-11T04:59:08.090647+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:08.090647+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5720,"log_dups_size":0,"ondisk_log_size":5720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822560,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5718,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5708","reported_seq":5786,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:09.446491+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:09.446491+0000","last_peered":"2024-09-11T04:59:09.446491+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:09.446491+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732448,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3387}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5777","reported_seq":5840,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:06.840187+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:06.840187+0000","last_peered":"2024-09-11T04:59:06.840187+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:06.840187+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281312,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":46188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":749240512,"num_objects":91465,"num_object_clones":0,"num_object_copies":365860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91492,"num_write_kb":731818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1885,"num_bytes_recovered":15212544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1354637312,"data_stored":1354637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91519,"ondisk_log_size":91519,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712704,"kb_used_data":365872,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93659136,"statfs":{"total":96636764160,"available":95906955264,"internally_reserved":0,"allocated":374652928,"data_stored":374482118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,2,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709284,"kb_used_data":367124,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93662556,"statfs":{"total":96636764160,"available":95910457344,"internally_reserved":0,"allocated":375934976,"data_stored":375764166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,0,1,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432764,"kb_used_data":224812,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93939076,"statfs":{"total":96636764160,"available":96193613824,"internally_reserved":0,"allocated":230207488,"data_stored":230036678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":212922066},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702600,"kb_used_data":364152,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93669240,"statfs":{"total":96636764160,"available":95917301760,"internally_reserved":0,"allocated":372891648,"data_stored":372720838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373059584,"data_stored":373059584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230846464,"data_stored":230846464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376193024,"data_stored":376193024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374538240,"data_stored":374538240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:13.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 210 20 92905 92885 3.45503 2.23438 0.0831028 0.045209 2024-09-11T04:59:14.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 211 20 93183 93163 3.44895 2.17188 0.106956 0.0452887 2024-09-11T04:59:15.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 212 20 93463 93443 3.443 2.1875 0.11432 0.0453671 2024-09-11T04:59:15.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:16.408 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:16.408 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:16.437 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-11T04:59:15.687495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":759898304,"num_objects":92766,"num_object_clones":0,"num_object_copies":371064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92793,"num_write_kb":742226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92820,"ondisk_log_size":92820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2597156,"kb_used_data":1343716,"kb_used_omap":14,"kb_used_meta":1253361,"kb_avail":374890204,"statfs":{"total":386547056640,"available":383887568896,"internally_reserved":0,"allocated":1375965184,"data_stored":1375281944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1283442230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,15,10,14,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24731648,"num_objects":3019,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3019,"num_write_kb":24152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004700"},"pg_stats":[{"pgid":"1.f","version":"31'5748","reported_seq":5789,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.268842+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:13.268842+0000","last_peered":"2024-09-11T04:59:13.268842+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:13.268842+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47087616,"num_objects":5748,"num_object_clones":0,"num_object_copies":22992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5748,"num_write_kb":45984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5845","reported_seq":5907,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.953147+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:11.953147+0000","last_peered":"2024-09-11T04:59:11.953147+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:11.953147+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47846560,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5843,"num_write_kb":46740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5653","reported_seq":5728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.659955+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:14.659955+0000","last_peered":"2024-09-11T04:59:14.659955+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:14.659955+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46283344,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":45206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5702","reported_seq":5765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.817094+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:11.817094+0000","last_peered":"2024-09-11T04:59:11.817094+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:11.817094+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5702,"log_dups_size":0,"ondisk_log_size":5702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46675104,"num_objects":5698,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5700,"num_write_kb":45589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5801","reported_seq":5855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.193923+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:13.193923+0000","last_peered":"2024-09-11T04:59:13.193923+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:13.193923+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5801,"log_dups_size":0,"ondisk_log_size":5801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47494304,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5824","reported_seq":5865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.034857+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:13.034857+0000","last_peered":"2024-09-11T04:59:13.034857+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:13.034857+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":5824,"num_object_clones":0,"num_object_copies":23296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5824,"num_write_kb":46592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5836","reported_seq":5899,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.860887+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:11.860887+0000","last_peered":"2024-09-11T04:59:11.860887+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:11.860887+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47756448,"num_objects":5830,"num_object_clones":0,"num_object_copies":23320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5834,"num_write_kb":46652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5892","reported_seq":5954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.660978+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:11.660978+0000","last_peered":"2024-09-11T04:59:11.660978+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:11.660978+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48223392,"num_objects":5887,"num_object_clones":0,"num_object_copies":23548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5890,"num_write_kb":47108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5876","reported_seq":5931,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.267616+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:13.267616+0000","last_peered":"2024-09-11T04:59:13.267616+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:13.267616+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48064832,"num_objects":5868,"num_object_clones":0,"num_object_copies":23472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5845","reported_seq":5923,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.654305+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:14.654305+0000","last_peered":"2024-09-11T04:59:14.654305+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:14.654305+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47862944,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5843,"num_write_kb":46742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3525}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5802","reported_seq":5857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.270996+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:13.270996+0000","last_peered":"2024-09-11T04:59:13.270996+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:13.270996+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47484656,"num_objects":5797,"num_object_clones":0,"num_object_copies":23188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5717","reported_seq":9580,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.238896+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:13.238896+0000","last_peered":"2024-09-11T04:59:13.238896+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:13.238896+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46824016,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5394}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5842","reported_seq":5915,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.430171+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:14.430171+0000","last_peered":"2024-09-11T04:59:14.430171+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:14.430171+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47848016,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5841,"num_write_kb":46727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5803","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.190696+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:13.190696+0000","last_peered":"2024-09-11T04:59:13.190696+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:13.190696+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47502496,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3512}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5766","reported_seq":5844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.553603+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:14.553603+0000","last_peered":"2024-09-11T04:59:14.553603+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:14.553603+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207584,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5868","reported_seq":5931,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.814939+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:11.814939+0000","last_peered":"2024-09-11T04:59:11.814939+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:11.814939+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5868,"log_dups_size":0,"ondisk_log_size":5868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48026784,"num_objects":5863,"num_object_clones":0,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":46916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3552}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":759898304,"num_objects":92766,"num_object_clones":0,"num_object_copies":371064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92793,"num_write_kb":742226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1376792576,"data_stored":1376792576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92820,"ondisk_log_size":92820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705572,"kb_used_data":371220,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93666268,"statfs":{"total":96636764160,"available":95914258432,"internally_reserved":0,"allocated":380129280,"data_stored":379958470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,7,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718704,"kb_used_data":372384,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93653136,"statfs":{"total":96636764160,"available":95900811264,"internally_reserved":0,"allocated":381321216,"data_stored":381150406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460620,"kb_used_data":230460,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93911220,"statfs":{"total":96636764160,"available":96165089280,"internally_reserved":0,"allocated":235991040,"data_stored":235820230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235663058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712260,"kb_used_data":369652,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93659580,"statfs":{"total":96636764160,"available":95907409920,"internally_reserved":0,"allocated":378523648,"data_stored":378352838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350809807},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,1,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378605568,"data_stored":378605568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236503040,"data_stored":236503040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381636608,"data_stored":381636608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":380047360,"data_stored":380047360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:16.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:16.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 213 20 93708 93688 3.43582 1.91406 0.0897253 0.0454607 2024-09-11T04:59:17.054 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:17.055 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:17.087 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-11T04:59:15.687495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":759898304,"num_objects":92766,"num_object_clones":0,"num_object_copies":371064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92793,"num_write_kb":742226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92820,"ondisk_log_size":92820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2597156,"kb_used_data":1343716,"kb_used_omap":14,"kb_used_meta":1253361,"kb_avail":374890204,"statfs":{"total":386547056640,"available":383887568896,"internally_reserved":0,"allocated":1375965184,"data_stored":1375281944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1283442230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,15,10,14,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24731648,"num_objects":3019,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3019,"num_write_kb":24152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004700"},"pg_stats":[{"pgid":"1.f","version":"31'5748","reported_seq":5789,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.268842+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:13.268842+0000","last_peered":"2024-09-11T04:59:13.268842+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:13.268842+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47087616,"num_objects":5748,"num_object_clones":0,"num_object_copies":22992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5748,"num_write_kb":45984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5845","reported_seq":5907,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.953147+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:11.953147+0000","last_peered":"2024-09-11T04:59:11.953147+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:11.953147+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47846560,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5843,"num_write_kb":46740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5653","reported_seq":5728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.659955+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:14.659955+0000","last_peered":"2024-09-11T04:59:14.659955+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:14.659955+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46283344,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":45206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5702","reported_seq":5765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.817094+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:11.817094+0000","last_peered":"2024-09-11T04:59:11.817094+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:11.817094+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5702,"log_dups_size":0,"ondisk_log_size":5702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46675104,"num_objects":5698,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5700,"num_write_kb":45589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5801","reported_seq":5855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.193923+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:13.193923+0000","last_peered":"2024-09-11T04:59:13.193923+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:13.193923+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5801,"log_dups_size":0,"ondisk_log_size":5801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47494304,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5824","reported_seq":5865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.034857+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:13.034857+0000","last_peered":"2024-09-11T04:59:13.034857+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:13.034857+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":5824,"num_object_clones":0,"num_object_copies":23296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5824,"num_write_kb":46592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5836","reported_seq":5899,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.860887+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:11.860887+0000","last_peered":"2024-09-11T04:59:11.860887+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:11.860887+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47756448,"num_objects":5830,"num_object_clones":0,"num_object_copies":23320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5834,"num_write_kb":46652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5892","reported_seq":5954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.660978+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:11.660978+0000","last_peered":"2024-09-11T04:59:11.660978+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:11.660978+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48223392,"num_objects":5887,"num_object_clones":0,"num_object_copies":23548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5890,"num_write_kb":47108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5876","reported_seq":5931,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.267616+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:13.267616+0000","last_peered":"2024-09-11T04:59:13.267616+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:13.267616+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48064832,"num_objects":5868,"num_object_clones":0,"num_object_copies":23472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5845","reported_seq":5923,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.654305+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:14.654305+0000","last_peered":"2024-09-11T04:59:14.654305+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:14.654305+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47862944,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5843,"num_write_kb":46742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3525}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5802","reported_seq":5857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.270996+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:13.270996+0000","last_peered":"2024-09-11T04:59:13.270996+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:13.270996+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47484656,"num_objects":5797,"num_object_clones":0,"num_object_copies":23188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5717","reported_seq":9580,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.238896+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:13.238896+0000","last_peered":"2024-09-11T04:59:13.238896+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:13.238896+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46824016,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5394}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5842","reported_seq":5915,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.430171+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:14.430171+0000","last_peered":"2024-09-11T04:59:14.430171+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:14.430171+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47848016,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5841,"num_write_kb":46727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5803","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.190696+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:13.190696+0000","last_peered":"2024-09-11T04:59:13.190696+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:13.190696+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47502496,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3512}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5766","reported_seq":5844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.553603+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:14.553603+0000","last_peered":"2024-09-11T04:59:14.553603+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:14.553603+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207584,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5868","reported_seq":5931,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:11.814939+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:11.814939+0000","last_peered":"2024-09-11T04:59:11.814939+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:11.814939+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5868,"log_dups_size":0,"ondisk_log_size":5868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48026784,"num_objects":5863,"num_object_clones":0,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":46916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3552}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":759898304,"num_objects":92766,"num_object_clones":0,"num_object_copies":371064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92793,"num_write_kb":742226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1376792576,"data_stored":1376792576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92820,"ondisk_log_size":92820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705572,"kb_used_data":371220,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93666268,"statfs":{"total":96636764160,"available":95914258432,"internally_reserved":0,"allocated":380129280,"data_stored":379958470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,7,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718704,"kb_used_data":372384,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93653136,"statfs":{"total":96636764160,"available":95900811264,"internally_reserved":0,"allocated":381321216,"data_stored":381150406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460620,"kb_used_data":230460,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93911220,"statfs":{"total":96636764160,"available":96165089280,"internally_reserved":0,"allocated":235991040,"data_stored":235820230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235663058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712260,"kb_used_data":369652,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93659580,"statfs":{"total":96636764160,"available":95907409920,"internally_reserved":0,"allocated":378523648,"data_stored":378352838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350809807},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,1,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378605568,"data_stored":378605568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236503040,"data_stored":236503040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381636608,"data_stored":381636608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":380047360,"data_stored":380047360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:17.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:17.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5748", 'reported_seq': 5789, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:13.268842+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:13.268842+0000', 'last_peered': '2024-09-11T04:59:13.268842+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:13.268842+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5748, 'log_dups_size': 0, 'ondisk_log_size': 5748, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47087616, 'num_objects': 5748, 'num_object_clones': 0, 'num_object_copies': 22992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5748, 'num_write_kb': 45984, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3436}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:17.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:17.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5845", 'reported_seq': 5907, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:11.953147+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:11.953147+0000', 'last_peered': '2024-09-11T04:59:11.953147+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:11.953147+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5845, 'log_dups_size': 0, 'ondisk_log_size': 5845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47846560, 'num_objects': 5841, 'num_object_clones': 0, 'num_object_copies': 23364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5843, 'num_write_kb': 46740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3562}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:17.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5653", 'reported_seq': 5728, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:14.659955+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:14.659955+0000', 'last_peered': '2024-09-11T04:59:14.659955+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:14.659955+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5653, 'log_dups_size': 0, 'ondisk_log_size': 5653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46283344, 'num_objects': 5650, 'num_object_clones': 0, 'num_object_copies': 22600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5652, 'num_write_kb': 45206, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3430}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5702", 'reported_seq': 5765, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:11.817094+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:11.817094+0000', 'last_peered': '2024-09-11T04:59:11.817094+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:11.817094+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5702, 'log_dups_size': 0, 'ondisk_log_size': 5702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46675104, 'num_objects': 5698, 'num_object_clones': 0, 'num_object_copies': 22792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5700, 'num_write_kb': 45589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5801", 'reported_seq': 5855, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:13.193923+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:13.193923+0000', 'last_peered': '2024-09-11T04:59:13.193923+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:13.193923+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5801, 'log_dups_size': 0, 'ondisk_log_size': 5801, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47494304, 'num_objects': 5798, 'num_object_clones': 0, 'num_object_copies': 23192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5799, 'num_write_kb': 46389, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3548}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5824", 'reported_seq': 5865, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:13.034857+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:13.034857+0000', 'last_peered': '2024-09-11T04:59:13.034857+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:13.034857+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5824, 'log_dups_size': 0, 'ondisk_log_size': 5824, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47710208, 'num_objects': 5824, 'num_object_clones': 0, 'num_object_copies': 23296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5824, 'num_write_kb': 46592, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3503}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:17.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5836", 'reported_seq': 5899, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:11.860887+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:11.860887+0000', 'last_peered': '2024-09-11T04:59:11.860887+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:11.860887+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5836, 'log_dups_size': 0, 'ondisk_log_size': 5836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47756448, 'num_objects': 5830, 'num_object_clones': 0, 'num_object_copies': 23320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5834, 'num_write_kb': 46652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3513}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5892", 'reported_seq': 5954, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:11.660978+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:59:11.660978+0000', 'last_peered': '2024-09-11T04:59:11.660978+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:11.660978+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5892, 'log_dups_size': 0, 'ondisk_log_size': 5892, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48223392, 'num_objects': 5887, 'num_object_clones': 0, 'num_object_copies': 23548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5890, 'num_write_kb': 47108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5876", 'reported_seq': 5931, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:13.267616+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:13.267616+0000', 'last_peered': '2024-09-11T04:59:13.267616+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:13.267616+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5876, 'log_dups_size': 0, 'ondisk_log_size': 5876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48064832, 'num_objects': 5868, 'num_object_clones': 0, 'num_object_copies': 23472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5873, 'num_write_kb': 46961, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5845", 'reported_seq': 5923, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:14.654305+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:14.654305+0000', 'last_peered': '2024-09-11T04:59:14.654305+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:14.654305+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5845, 'log_dups_size': 0, 'ondisk_log_size': 5845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47862944, 'num_objects': 5843, 'num_object_clones': 0, 'num_object_copies': 23372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5843, 'num_write_kb': 46742, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3525}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5802", 'reported_seq': 5857, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:13.270996+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:13.270996+0000', 'last_peered': '2024-09-11T04:59:13.270996+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:13.270996+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5802, 'log_dups_size': 0, 'ondisk_log_size': 5802, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47484656, 'num_objects': 5797, 'num_object_clones': 0, 'num_object_copies': 23188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5799, 'num_write_kb': 46380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:59:17.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5717", 'reported_seq': 9580, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:13.238896+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:59:13.238896+0000', 'last_peered': '2024-09-11T04:59:13.238896+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:59:13.238896+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5717, 'log_dups_size': 0, 'ondisk_log_size': 5717, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46824016, 'num_objects': 5716, 'num_object_clones': 0, 'num_object_copies': 22864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5716, 'num_write_kb': 45727, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1915, 'num_bytes_recovered': 15671296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5394}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:17.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:17.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5842", 'reported_seq': 5915, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:14.430171+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:14.430171+0000', 'last_peered': '2024-09-11T04:59:14.430171+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:14.430171+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5842, 'log_dups_size': 0, 'ondisk_log_size': 5842, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47848016, 'num_objects': 5841, 'num_object_clones': 0, 'num_object_copies': 23364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5841, 'num_write_kb': 46727, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3509}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:17.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:17.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5803", 'reported_seq': 5853, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:13.190696+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:13.190696+0000', 'last_peered': '2024-09-11T04:59:13.190696+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:13.190696+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5803, 'log_dups_size': 0, 'ondisk_log_size': 5803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47502496, 'num_objects': 5799, 'num_object_clones': 0, 'num_object_copies': 23196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5801, 'num_write_kb': 46404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:17.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:17.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5766", 'reported_seq': 5844, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:14.553603+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:14.553603+0000', 'last_peered': '2024-09-11T04:59:14.553603+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:14.553603+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5766, 'log_dups_size': 0, 'ondisk_log_size': 5766, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47207584, 'num_objects': 5763, 'num_object_clones': 0, 'num_object_copies': 23052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5764, 'num_write_kb': 46109, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:17.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:17.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5868", 'reported_seq': 5931, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:11.814939+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:11.814939+0000', 'last_peered': '2024-09-11T04:59:11.814939+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:11.814939+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5868, 'log_dups_size': 0, 'ondisk_log_size': 5868, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48026784, 'num_objects': 5863, 'num_object_clones': 0, 'num_object_copies': 23452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5866, 'num_write_kb': 46916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:17.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:17.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 214 20 93955 93935 3.42878 1.92969 0.0922082 0.0455584 2024-09-11T04:59:17.849 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:17.849 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34969/372800 objects degraded (9.380%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":284,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93200,"data_bytes":763453632,"bytes_used":2669142016,"bytes_avail":383877914624,"bytes_total":386547056640,"degraded_objects":34969,"degraded_total":372800,"degraded_ratio":0.093800965665236047,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":79844,"recovering_keys_per_sec":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"write_bytes_sec":2112114,"read_op_per_sec":0,"write_op_per_sec":257},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:17.878 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:17.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:18.468 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:18.499 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-11T04:59:17.688023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":763453632,"num_objects":93200,"num_object_clones":0,"num_object_copies":372800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93227,"num_write_kb":745698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93254,"ondisk_log_size":93254,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2606584,"kb_used_data":1348984,"kb_used_omap":14,"kb_used_meta":1257521,"kb_avail":374880776,"statfs":{"total":386547056640,"available":383877914624,"internally_reserved":0,"allocated":1381359616,"data_stored":1380676376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1287702070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,11,13,15,15,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25354240,"num_objects":3095,"num_object_clones":0,"num_object_copies":12380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3095,"num_write_kb":24760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004197"},"pg_stats":[{"pgid":"1.f","version":"31'5748","reported_seq":5789,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.268842+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:13.268842+0000","last_peered":"2024-09-11T04:59:13.268842+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:13.268842+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47087616,"num_objects":5748,"num_object_clones":0,"num_object_copies":22992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5748,"num_write_kb":45984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5932","reported_seq":5994,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.636648+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:16.636648+0000","last_peered":"2024-09-11T04:59:16.636648+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:16.636648+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48559264,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":47436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5653","reported_seq":5728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.659955+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:14.659955+0000","last_peered":"2024-09-11T04:59:14.659955+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:14.659955+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46283344,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":45206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5775","reported_seq":5838,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.924210+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:16.924210+0000","last_peered":"2024-09-11T04:59:16.924210+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:16.924210+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47273120,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5801","reported_seq":5855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.193923+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:13.193923+0000","last_peered":"2024-09-11T04:59:13.193923+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:13.193923+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5801,"log_dups_size":0,"ondisk_log_size":5801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47494304,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5824","reported_seq":5865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.034857+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:13.034857+0000","last_peered":"2024-09-11T04:59:13.034857+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:13.034857+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":5824,"num_object_clones":0,"num_object_copies":23296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5824,"num_write_kb":46592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5933","reported_seq":5996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.873967+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:16.873967+0000","last_peered":"2024-09-11T04:59:16.873967+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:16.873967+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48551072,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5931,"num_write_kb":47428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3610}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5978","reported_seq":6040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.816272+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:16.816272+0000","last_peered":"2024-09-11T04:59:16.816272+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:16.816272+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48927904,"num_objects":5973,"num_object_clones":0,"num_object_copies":23892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5976,"num_write_kb":47796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5876","reported_seq":5931,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.267616+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:13.267616+0000","last_peered":"2024-09-11T04:59:13.267616+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:13.267616+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48064832,"num_objects":5868,"num_object_clones":0,"num_object_copies":23472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5845","reported_seq":5923,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.654305+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:14.654305+0000","last_peered":"2024-09-11T04:59:14.654305+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:14.654305+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47862944,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5843,"num_write_kb":46742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3525}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5802","reported_seq":5857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.270996+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:13.270996+0000","last_peered":"2024-09-11T04:59:13.270996+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:13.270996+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47484656,"num_objects":5797,"num_object_clones":0,"num_object_copies":23188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5717","reported_seq":9580,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.238896+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:13.238896+0000","last_peered":"2024-09-11T04:59:13.238896+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:13.238896+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46824016,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":45727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5394}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5842","reported_seq":5915,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.430171+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:14.430171+0000","last_peered":"2024-09-11T04:59:14.430171+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:14.430171+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47848016,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5841,"num_write_kb":46727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5803","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:13.190696+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:13.190696+0000","last_peered":"2024-09-11T04:59:13.190696+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:13.190696+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47502496,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3512}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5766","reported_seq":5844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:14.553603+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:14.553603+0000","last_peered":"2024-09-11T04:59:14.553603+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:14.553603+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47207584,"num_objects":5763,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5959","reported_seq":6022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.899866+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:16.899866+0000","last_peered":"2024-09-11T04:59:16.899866+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:16.899866+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48772256,"num_objects":5954,"num_object_clones":0,"num_object_copies":23816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":47644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":763453632,"num_objects":93200,"num_object_clones":0,"num_object_copies":372800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93227,"num_write_kb":745698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":15671296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1382264832,"data_stored":1382264832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93254,"ondisk_log_size":93254,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705572,"kb_used_data":371220,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93666268,"statfs":{"total":96636764160,"available":95914258432,"internally_reserved":0,"allocated":380129280,"data_stored":379958470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,7,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718704,"kb_used_data":372384,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93653136,"statfs":{"total":96636764160,"available":95900811264,"internally_reserved":0,"allocated":381321216,"data_stored":381150406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460620,"kb_used_data":230460,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93911220,"statfs":{"total":96636764160,"available":96165089280,"internally_reserved":0,"allocated":235991040,"data_stored":235820230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235663058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721688,"kb_used_data":374920,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93650152,"statfs":{"total":96636764160,"available":95897755648,"internally_reserved":0,"allocated":383918080,"data_stored":383747270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384077824,"data_stored":384077824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236503040,"data_stored":236503040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381636608,"data_stored":381636608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":380047360,"data_stored":380047360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:18.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 215 20 94212 94192 3.42217 2.00781 0.0394508 0.0456466 2024-09-11T04:59:19.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 216 20 94498 94478 3.41667 2.23438 0.058853 0.0457177 2024-09-11T04:59:20.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 217 20 94783 94763 3.41119 2.22656 0.0627328 0.0457936 2024-09-11T04:59:21.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:21.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 218 20 95059 95039 3.40543 2.15625 0.129699 0.0458674 2024-09-11T04:59:22.151 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:22.189 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-11T04:59:21.689644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":770957504,"num_objects":94116,"num_object_clones":0,"num_object_copies":376464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94143,"num_write_kb":753026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":16162816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94170,"ondisk_log_size":94170,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2618448,"kb_used_data":1365004,"kb_used_omap":14,"kb_used_meta":1253361,"kb_avail":374868912,"statfs":{"total":386547056640,"available":383865765888,"internally_reserved":0,"allocated":1397764096,"data_stored":1397080856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1283442230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,4,3,7,12,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21716992,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":21208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004204"},"pg_stats":[{"pgid":"1.f","version":"31'5845","reported_seq":5887,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.031719+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:18.031719+0000","last_peered":"2024-09-11T04:59:18.031719+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:18.031719+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47882240,"num_objects":5845,"num_object_clones":0,"num_object_copies":23380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5845,"num_write_kb":46760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5932","reported_seq":5994,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.636648+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:16.636648+0000","last_peered":"2024-09-11T04:59:16.636648+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:16.636648+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48559264,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":47436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5731","reported_seq":5806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.589976+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:19.589976+0000","last_peered":"2024-09-11T04:59:19.589976+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:19.589976+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46922320,"num_objects":5728,"num_object_clones":0,"num_object_copies":22912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5730,"num_write_kb":45830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3508}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5775","reported_seq":5838,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.924210+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:16.924210+0000","last_peered":"2024-09-11T04:59:16.924210+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:16.924210+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47273120,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5887","reported_seq":5941,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.254698+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:18.254698+0000","last_peered":"2024-09-11T04:59:18.254698+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:18.254698+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48198816,"num_objects":5884,"num_object_clones":0,"num_object_copies":23536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5885,"num_write_kb":47077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5906","reported_seq":5947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.286430+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:18.286430+0000","last_peered":"2024-09-11T04:59:18.286430+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:18.286430+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5906,"log_dups_size":0,"ondisk_log_size":5906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48381952,"num_objects":5906,"num_object_clones":0,"num_object_copies":23624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5906,"num_write_kb":47248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5933","reported_seq":5996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.873967+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:16.873967+0000","last_peered":"2024-09-11T04:59:16.873967+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:16.873967+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48551072,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5931,"num_write_kb":47428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3610}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5978","reported_seq":6040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.816272+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:16.816272+0000","last_peered":"2024-09-11T04:59:16.816272+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:16.816272+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48927904,"num_objects":5973,"num_object_clones":0,"num_object_copies":23892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5976,"num_write_kb":47796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5981","reported_seq":6036,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.280255+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:18.280255+0000","last_peered":"2024-09-11T04:59:18.280255+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:18.280255+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5981,"log_dups_size":0,"ondisk_log_size":5981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48924992,"num_objects":5973,"num_object_clones":0,"num_object_copies":23892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5978,"num_write_kb":47801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5912","reported_seq":5990,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.671872+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:19.671872+0000","last_peered":"2024-09-11T04:59:19.671872+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:19.671872+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48411808,"num_objects":5910,"num_object_clones":0,"num_object_copies":23640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":47278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3592}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5881","reported_seq":5937,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.251627+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:18.251627+0000","last_peered":"2024-09-11T04:59:18.251627+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:18.251627+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5881,"log_dups_size":0,"ondisk_log_size":5881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48131824,"num_objects":5876,"num_object_clones":0,"num_object_copies":23504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5878,"num_write_kb":47012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5789","reported_seq":9774,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.272069+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:18.272069+0000","last_peered":"2024-09-11T04:59:18.272069+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:18.272069+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47413840,"num_objects":5788,"num_object_clones":0,"num_object_copies":23152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":46303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1975,"num_bytes_recovered":16162816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5526}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5940","reported_seq":6014,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.639505+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:19.639505+0000","last_peered":"2024-09-11T04:59:19.639505+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:19.639505+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48650832,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5939,"num_write_kb":47511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5879","reported_seq":5929,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.176265+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:18.176265+0000","last_peered":"2024-09-11T04:59:18.176265+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:18.176265+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5879,"log_dups_size":0,"ondisk_log_size":5879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48125088,"num_objects":5875,"num_object_clones":0,"num_object_copies":23500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5877,"num_write_kb":47012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3588}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5842","reported_seq":5920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.590504+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:19.590504+0000","last_peered":"2024-09-11T04:59:19.590504+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:19.590504+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47830176,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3521}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5959","reported_seq":6022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.899866+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:16.899866+0000","last_peered":"2024-09-11T04:59:16.899866+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:16.899866+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48772256,"num_objects":5954,"num_object_clones":0,"num_object_copies":23816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":47644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":770957504,"num_objects":94116,"num_object_clones":0,"num_object_copies":376464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94143,"num_write_kb":753026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":16162816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1398800384,"data_stored":1398800384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94170,"ondisk_log_size":94170,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714964,"kb_used_data":376452,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93656876,"statfs":{"total":96636764160,"available":95904641024,"internally_reserved":0,"allocated":385486848,"data_stored":385316038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":346615504},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,1,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711540,"kb_used_data":377696,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93660300,"statfs":{"total":96636764160,"available":95908147200,"internally_reserved":0,"allocated":386760704,"data_stored":386589894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470256,"kb_used_data":235936,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93901584,"statfs":{"total":96636764160,"available":96155222016,"internally_reserved":0,"allocated":241598464,"data_stored":241427654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239922898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721688,"kb_used_data":374920,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93650152,"statfs":{"total":96636764160,"available":95897755648,"internally_reserved":0,"allocated":383918080,"data_stored":383747270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384077824,"data_stored":384077824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242245632,"data_stored":242245632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387039232,"data_stored":387039232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385437696,"data_stored":385437696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:22.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:22.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 219 20 95312 95292 3.39891 1.97656 0.0404691 0.0459539 2024-09-11T04:59:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:22.816 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:22.839 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-11T04:59:21.689644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":770957504,"num_objects":94116,"num_object_clones":0,"num_object_copies":376464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94143,"num_write_kb":753026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":16162816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94170,"ondisk_log_size":94170,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2618448,"kb_used_data":1365004,"kb_used_omap":14,"kb_used_meta":1253361,"kb_avail":374868912,"statfs":{"total":386547056640,"available":383865765888,"internally_reserved":0,"allocated":1397764096,"data_stored":1397080856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1283442230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,4,3,7,12,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21716992,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":21208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004204"},"pg_stats":[{"pgid":"1.f","version":"31'5845","reported_seq":5887,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.031719+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:18.031719+0000","last_peered":"2024-09-11T04:59:18.031719+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:18.031719+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47882240,"num_objects":5845,"num_object_clones":0,"num_object_copies":23380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5845,"num_write_kb":46760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5932","reported_seq":5994,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.636648+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:16.636648+0000","last_peered":"2024-09-11T04:59:16.636648+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:16.636648+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48559264,"num_objects":5928,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":47436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5731","reported_seq":5806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.589976+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:19.589976+0000","last_peered":"2024-09-11T04:59:19.589976+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:19.589976+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46922320,"num_objects":5728,"num_object_clones":0,"num_object_copies":22912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5730,"num_write_kb":45830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3508}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5775","reported_seq":5838,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.924210+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:16.924210+0000","last_peered":"2024-09-11T04:59:16.924210+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:16.924210+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47273120,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5887","reported_seq":5941,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.254698+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:18.254698+0000","last_peered":"2024-09-11T04:59:18.254698+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:18.254698+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48198816,"num_objects":5884,"num_object_clones":0,"num_object_copies":23536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5885,"num_write_kb":47077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5906","reported_seq":5947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.286430+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:18.286430+0000","last_peered":"2024-09-11T04:59:18.286430+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:18.286430+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5906,"log_dups_size":0,"ondisk_log_size":5906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48381952,"num_objects":5906,"num_object_clones":0,"num_object_copies":23624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5906,"num_write_kb":47248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5933","reported_seq":5996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.873967+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:16.873967+0000","last_peered":"2024-09-11T04:59:16.873967+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:16.873967+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48551072,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5931,"num_write_kb":47428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3610}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5978","reported_seq":6040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.816272+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:16.816272+0000","last_peered":"2024-09-11T04:59:16.816272+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:16.816272+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48927904,"num_objects":5973,"num_object_clones":0,"num_object_copies":23892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5976,"num_write_kb":47796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5981","reported_seq":6036,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.280255+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:18.280255+0000","last_peered":"2024-09-11T04:59:18.280255+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:18.280255+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5981,"log_dups_size":0,"ondisk_log_size":5981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48924992,"num_objects":5973,"num_object_clones":0,"num_object_copies":23892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5978,"num_write_kb":47801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5912","reported_seq":5990,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.671872+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:19.671872+0000","last_peered":"2024-09-11T04:59:19.671872+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:19.671872+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48411808,"num_objects":5910,"num_object_clones":0,"num_object_copies":23640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":47278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3592}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5881","reported_seq":5937,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.251627+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:18.251627+0000","last_peered":"2024-09-11T04:59:18.251627+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:18.251627+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5881,"log_dups_size":0,"ondisk_log_size":5881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48131824,"num_objects":5876,"num_object_clones":0,"num_object_copies":23504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5878,"num_write_kb":47012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5789","reported_seq":9774,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.272069+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:18.272069+0000","last_peered":"2024-09-11T04:59:18.272069+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:18.272069+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47413840,"num_objects":5788,"num_object_clones":0,"num_object_copies":23152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":46303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1975,"num_bytes_recovered":16162816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5526}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5940","reported_seq":6014,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.639505+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:19.639505+0000","last_peered":"2024-09-11T04:59:19.639505+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:19.639505+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48650832,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5939,"num_write_kb":47511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5879","reported_seq":5929,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:18.176265+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:18.176265+0000","last_peered":"2024-09-11T04:59:18.176265+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:18.176265+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5879,"log_dups_size":0,"ondisk_log_size":5879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48125088,"num_objects":5875,"num_object_clones":0,"num_object_copies":23500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5877,"num_write_kb":47012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3588}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5842","reported_seq":5920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.590504+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:19.590504+0000","last_peered":"2024-09-11T04:59:19.590504+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:19.590504+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47830176,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3521}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5959","reported_seq":6022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:16.899866+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:16.899866+0000","last_peered":"2024-09-11T04:59:16.899866+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:16.899866+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48772256,"num_objects":5954,"num_object_clones":0,"num_object_copies":23816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":47644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":770957504,"num_objects":94116,"num_object_clones":0,"num_object_copies":376464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94143,"num_write_kb":753026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":16162816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1398800384,"data_stored":1398800384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94170,"ondisk_log_size":94170,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714964,"kb_used_data":376452,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93656876,"statfs":{"total":96636764160,"available":95904641024,"internally_reserved":0,"allocated":385486848,"data_stored":385316038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":346615504},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,1,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711540,"kb_used_data":377696,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93660300,"statfs":{"total":96636764160,"available":95908147200,"internally_reserved":0,"allocated":386760704,"data_stored":386589894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470256,"kb_used_data":235936,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93901584,"statfs":{"total":96636764160,"available":96155222016,"internally_reserved":0,"allocated":241598464,"data_stored":241427654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239922898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721688,"kb_used_data":374920,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93650152,"statfs":{"total":96636764160,"available":95897755648,"internally_reserved":0,"allocated":383918080,"data_stored":383747270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384077824,"data_stored":384077824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242245632,"data_stored":242245632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387039232,"data_stored":387039232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385437696,"data_stored":385437696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:22.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:22.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5845", 'reported_seq': 5887, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:18.031719+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:18.031719+0000', 'last_peered': '2024-09-11T04:59:18.031719+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:18.031719+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5845, 'log_dups_size': 0, 'ondisk_log_size': 5845, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47882240, 'num_objects': 5845, 'num_object_clones': 0, 'num_object_copies': 23380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5845, 'num_write_kb': 46760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3533}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5932", 'reported_seq': 5994, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:16.636648+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:16.636648+0000', 'last_peered': '2024-09-11T04:59:16.636648+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:16.636648+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5932, 'log_dups_size': 0, 'ondisk_log_size': 5932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48559264, 'num_objects': 5928, 'num_object_clones': 0, 'num_object_copies': 23712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5930, 'num_write_kb': 47436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5731", 'reported_seq': 5806, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:19.589976+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:19.589976+0000', 'last_peered': '2024-09-11T04:59:19.589976+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:19.589976+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5731, 'log_dups_size': 0, 'ondisk_log_size': 5731, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46922320, 'num_objects': 5728, 'num_object_clones': 0, 'num_object_copies': 22912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5730, 'num_write_kb': 45830, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3508}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5775", 'reported_seq': 5838, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:16.924210+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:16.924210+0000', 'last_peered': '2024-09-11T04:59:16.924210+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:16.924210+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5775, 'log_dups_size': 0, 'ondisk_log_size': 5775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47273120, 'num_objects': 5771, 'num_object_clones': 0, 'num_object_copies': 23084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5773, 'num_write_kb': 46173, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3493}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5887", 'reported_seq': 5941, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:18.254698+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:18.254698+0000', 'last_peered': '2024-09-11T04:59:18.254698+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:18.254698+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5887, 'log_dups_size': 0, 'ondisk_log_size': 5887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48198816, 'num_objects': 5884, 'num_object_clones': 0, 'num_object_copies': 23536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5885, 'num_write_kb': 47077, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3634}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5906", 'reported_seq': 5947, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:18.286430+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:18.286430+0000', 'last_peered': '2024-09-11T04:59:18.286430+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:18.286430+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5906, 'log_dups_size': 0, 'ondisk_log_size': 5906, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48381952, 'num_objects': 5906, 'num_object_clones': 0, 'num_object_copies': 23624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5906, 'num_write_kb': 47248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3585}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:22.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5933", 'reported_seq': 5996, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:16.873967+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:16.873967+0000', 'last_peered': '2024-09-11T04:59:16.873967+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:16.873967+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5933, 'log_dups_size': 0, 'ondisk_log_size': 5933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48551072, 'num_objects': 5927, 'num_object_clones': 0, 'num_object_copies': 23708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5931, 'num_write_kb': 47428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3610}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5978", 'reported_seq': 6040, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:16.816272+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:59:16.816272+0000', 'last_peered': '2024-09-11T04:59:16.816272+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:16.816272+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5978, 'log_dups_size': 0, 'ondisk_log_size': 5978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48927904, 'num_objects': 5973, 'num_object_clones': 0, 'num_object_copies': 23892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5976, 'num_write_kb': 47796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5981", 'reported_seq': 6036, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:18.280255+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:18.280255+0000', 'last_peered': '2024-09-11T04:59:18.280255+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:18.280255+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5981, 'log_dups_size': 0, 'ondisk_log_size': 5981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48924992, 'num_objects': 5973, 'num_object_clones': 0, 'num_object_copies': 23892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5978, 'num_write_kb': 47801, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3579}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5912", 'reported_seq': 5990, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:19.671872+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:19.671872+0000', 'last_peered': '2024-09-11T04:59:19.671872+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:19.671872+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5912, 'log_dups_size': 0, 'ondisk_log_size': 5912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48411808, 'num_objects': 5910, 'num_object_clones': 0, 'num_object_copies': 23640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5910, 'num_write_kb': 47278, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3592}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5881", 'reported_seq': 5937, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:18.251627+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:18.251627+0000', 'last_peered': '2024-09-11T04:59:18.251627+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:18.251627+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5881, 'log_dups_size': 0, 'ondisk_log_size': 5881, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48131824, 'num_objects': 5876, 'num_object_clones': 0, 'num_object_copies': 23504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5878, 'num_write_kb': 47012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3553}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:22.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5789", 'reported_seq': 9774, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:18.272069+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:59:18.272069+0000', 'last_peered': '2024-09-11T04:59:18.272069+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:59:18.272069+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5789, 'log_dups_size': 0, 'ondisk_log_size': 5789, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47413840, 'num_objects': 5788, 'num_object_clones': 0, 'num_object_copies': 23152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5788, 'num_write_kb': 46303, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1975, 'num_bytes_recovered': 16162816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5940", 'reported_seq': 6014, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:19.639505+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:19.639505+0000', 'last_peered': '2024-09-11T04:59:19.639505+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:19.639505+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5940, 'log_dups_size': 0, 'ondisk_log_size': 5940, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48650832, 'num_objects': 5939, 'num_object_clones': 0, 'num_object_copies': 23756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5939, 'num_write_kb': 47511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3607}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5879", 'reported_seq': 5929, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:18.176265+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:18.176265+0000', 'last_peered': '2024-09-11T04:59:18.176265+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:18.176265+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5879, 'log_dups_size': 0, 'ondisk_log_size': 5879, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48125088, 'num_objects': 5875, 'num_object_clones': 0, 'num_object_copies': 23500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5877, 'num_write_kb': 47012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3588}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5842", 'reported_seq': 5920, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:19.590504+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:19.590504+0000', 'last_peered': '2024-09-11T04:59:19.590504+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:19.590504+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5842, 'log_dups_size': 0, 'ondisk_log_size': 5842, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47830176, 'num_objects': 5839, 'num_object_clones': 0, 'num_object_copies': 23356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5840, 'num_write_kb': 46717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:22.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5959", 'reported_seq': 6022, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:16.899866+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:16.899866+0000', 'last_peered': '2024-09-11T04:59:16.899866+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:16.899866+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5959, 'log_dups_size': 0, 'ondisk_log_size': 5959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48772256, 'num_objects': 5954, 'num_object_clones': 0, 'num_object_copies': 23816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5957, 'num_write_kb': 47644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3643}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:22.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:23.582 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34909/376464 objects degraded (9.273%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":290,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94116,"data_bytes":770957504,"bytes_used":2681290752,"bytes_avail":383865765888,"bytes_total":386547056640,"degraded_objects":34909,"degraded_total":376464,"degraded_ratio":0.092728654001445018,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":79161,"recovering_keys_per_sec":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"write_bytes_sec":1809115,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:23.605 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:23.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:23.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:59:23.715244+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0460469 2024-09-11T04:59:23.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:59:23.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 220 20 95556 95536 3.39212 1.90625 0.0888672 0.0460469 2024-09-11T04:59:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:24.255 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:24.288 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-11T04:59:23.690841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":779296960,"num_objects":95134,"num_object_clones":0,"num_object_copies":380536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95161,"num_write_kb":761170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95188,"ondisk_log_size":95188,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2630860,"kb_used_data":1381576,"kb_used_omap":14,"kb_used_meta":1249201,"kb_avail":374856500,"statfs":{"total":386547056640,"available":383853056000,"internally_reserved":0,"allocated":1414733824,"data_stored":1414050584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1279182390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,10,6,11,9,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30056448,"num_objects":3669,"num_object_clones":0,"num_object_copies":14676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3669,"num_write_kb":29352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004676"},"pg_stats":[{"pgid":"1.f","version":"31'5926","reported_seq":5968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.305868+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:23.305868+0000","last_peered":"2024-09-11T04:59:23.305868+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:23.305868+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48545792,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":47408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6013","reported_seq":6075,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:21.904229+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:21.904229+0000","last_peered":"2024-09-11T04:59:21.904229+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:21.904229+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6013,"log_dups_size":0,"ondisk_log_size":6013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49222816,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6011,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5731","reported_seq":5806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.589976+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:19.589976+0000","last_peered":"2024-09-11T04:59:19.589976+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:19.589976+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46922320,"num_objects":5728,"num_object_clones":0,"num_object_copies":22912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5730,"num_write_kb":45830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3508}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5870","reported_seq":5932,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:21.884468+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:21.884468+0000","last_peered":"2024-09-11T04:59:21.884468+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:21.884468+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5870,"log_dups_size":0,"ondisk_log_size":5870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48051360,"num_objects":5866,"num_object_clones":0,"num_object_copies":23464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5868,"num_write_kb":46933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3588}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5952","reported_seq":6006,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.252877+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:23.252877+0000","last_peered":"2024-09-11T04:59:23.252877+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:23.252877+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731296,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":47597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5995","reported_seq":6035,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.305656+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:23.305656+0000","last_peered":"2024-09-11T04:59:23.305656+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:23.305656+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5995,"log_dups_size":0,"ondisk_log_size":5995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49111040,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6022","reported_seq":6084,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:21.948524+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:21.948524+0000","last_peered":"2024-09-11T04:59:21.948524+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:21.948524+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6022,"log_dups_size":0,"ondisk_log_size":6022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49280160,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6020,"num_write_kb":48140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3699}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6075","reported_seq":6137,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:21.959997+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:21.959997+0000","last_peered":"2024-09-11T04:59:21.959997+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:21.959997+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49722528,"num_objects":6070,"num_object_clones":0,"num_object_copies":24280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6073,"num_write_kb":48572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6066","reported_seq":6122,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.311830+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:23.311830+0000","last_peered":"2024-09-11T04:59:23.311830+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:23.311830+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49621312,"num_objects":6058,"num_object_clones":0,"num_object_copies":24232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6063,"num_write_kb":48481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5912","reported_seq":5990,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.671872+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:19.671872+0000","last_peered":"2024-09-11T04:59:19.671872+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:19.671872+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48411808,"num_objects":5910,"num_object_clones":0,"num_object_copies":23640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":47278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3592}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5976","reported_seq":6032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.364717+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:23.364717+0000","last_peered":"2024-09-11T04:59:23.364717+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:23.364717+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5976,"log_dups_size":0,"ondisk_log_size":5976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48910064,"num_objects":5971,"num_object_clones":0,"num_object_copies":23884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5973,"num_write_kb":47772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5874","reported_seq":10003,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.301583+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:23.301583+0000","last_peered":"2024-09-11T04:59:23.301583+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:23.301583+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5874,"log_dups_size":0,"ondisk_log_size":5874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48110160,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5940","reported_seq":6014,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.639505+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:19.639505+0000","last_peered":"2024-09-11T04:59:19.639505+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:19.639505+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48650832,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5939,"num_write_kb":47511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5953","reported_seq":6003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:22.993478+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:22.993478+0000","last_peered":"2024-09-11T04:59:22.993478+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:22.993478+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731296,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5842","reported_seq":5920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:19.590504+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:19.590504+0000","last_peered":"2024-09-11T04:59:19.590504+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:19.590504+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47830176,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3521}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6041","reported_seq":6104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:21.856458+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:21.856458+0000","last_peered":"2024-09-11T04:59:21.856458+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:21.856458+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6041,"log_dups_size":0,"ondisk_log_size":6041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49444000,"num_objects":6036,"num_object_clones":0,"num_object_copies":24144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":48300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3725}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":779296960,"num_objects":95134,"num_object_clones":0,"num_object_copies":380536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95161,"num_write_kb":761170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1415962624,"data_stored":1415962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95188,"ondisk_log_size":95188,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724644,"kb_used_data":381972,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93647196,"statfs":{"total":96636764160,"available":95894728704,"internally_reserved":0,"allocated":391139328,"data_stored":390968518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,3,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711540,"kb_used_data":377696,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93660300,"statfs":{"total":96636764160,"available":95908147200,"internally_reserved":0,"allocated":386760704,"data_stored":386589894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480096,"kb_used_data":241616,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93891744,"statfs":{"total":96636764160,"available":96145145856,"internally_reserved":0,"allocated":247414784,"data_stored":247243974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244182738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714580,"kb_used_data":380292,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93657260,"statfs":{"total":96636764160,"available":95905034240,"internally_reserved":0,"allocated":389419008,"data_stored":389248198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342290127},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389685248,"data_stored":389685248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248164352,"data_stored":248164352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387039232,"data_stored":387039232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391073792,"data_stored":391073792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:24.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 221 20 95816 95796 3.38596 2.03125 0.0525918 0.0461349 2024-09-11T04:59:25.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 222 20 96099 96079 3.38067 2.21094 0.0470582 0.046207 2024-09-11T04:59:26.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 223 20 96382 96362 3.37542 2.21094 0.031872 0.0462769 2024-09-11T04:59:27.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:27.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 224 20 96647 96627 3.3696 2.07031 0.0853921 0.0463604 2024-09-11T04:59:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:27.913 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:27.939 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-11T04:59:27.692514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":785268928,"num_objects":95863,"num_object_clones":0,"num_object_copies":383452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95890,"num_write_kb":767002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95917,"ondisk_log_size":95917,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2649708,"kb_used_data":1392108,"kb_used_omap":14,"kb_used_meta":1257521,"kb_avail":374837652,"statfs":{"total":386547056640,"available":383833755648,"internally_reserved":0,"allocated":1425518592,"data_stored":1424835352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1287702070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,5,6,16,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25370624,"num_objects":3097,"num_object_clones":0,"num_object_copies":12388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":24776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1081344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005019"},"pg_stats":[{"pgid":"1.f","version":"31'5926","reported_seq":5968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.305868+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:23.305868+0000","last_peered":"2024-09-11T04:59:23.305868+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:23.305868+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48545792,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":47408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6112","reported_seq":6174,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.973110+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:26.973110+0000","last_peered":"2024-09-11T04:59:26.973110+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:26.973110+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50033824,"num_objects":6108,"num_object_clones":0,"num_object_copies":24432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6110,"num_write_kb":48876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5794","reported_seq":5869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.320778+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:24.320778+0000","last_peered":"2024-09-11T04:59:24.320778+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:24.320778+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47438416,"num_objects":5791,"num_object_clones":0,"num_object_copies":23164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":46334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3571}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5938","reported_seq":6002,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:27.024339+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:27.024339+0000","last_peered":"2024-09-11T04:59:27.024339+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:27.024339+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48608416,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3656}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5952","reported_seq":6006,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.252877+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:23.252877+0000","last_peered":"2024-09-11T04:59:23.252877+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:23.252877+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731296,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":47597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5995","reported_seq":6035,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.305656+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:23.305656+0000","last_peered":"2024-09-11T04:59:23.305656+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:23.305656+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5995,"log_dups_size":0,"ondisk_log_size":5995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49111040,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6117","reported_seq":6180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.873699+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:26.873699+0000","last_peered":"2024-09-11T04:59:26.873699+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:26.873699+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6117,"log_dups_size":0,"ondisk_log_size":6117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058400,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6115,"num_write_kb":48900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6139","reported_seq":6201,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.782139+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:26.782139+0000","last_peered":"2024-09-11T04:59:26.782139+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:26.782139+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246816,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":49084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6066","reported_seq":6122,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.311830+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:23.311830+0000","last_peered":"2024-09-11T04:59:23.311830+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:23.311830+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49621312,"num_objects":6058,"num_object_clones":0,"num_object_copies":24232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6063,"num_write_kb":48481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5988","reported_seq":6066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.682259+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:24.682259+0000","last_peered":"2024-09-11T04:59:24.682259+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:24.682259+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49034400,"num_objects":5986,"num_object_clones":0,"num_object_copies":23944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":47886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3668}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5976","reported_seq":6032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.364717+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:23.364717+0000","last_peered":"2024-09-11T04:59:23.364717+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:23.364717+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5976,"log_dups_size":0,"ondisk_log_size":5976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48910064,"num_objects":5971,"num_object_clones":0,"num_object_copies":23884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5973,"num_write_kb":47772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5874","reported_seq":10003,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.301583+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:23.301583+0000","last_peered":"2024-09-11T04:59:23.301583+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:23.301583+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5874,"log_dups_size":0,"ondisk_log_size":5874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48110160,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6040","reported_seq":6114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.630445+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:24.630445+0000","last_peered":"2024-09-11T04:59:24.630445+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:24.630445+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49470032,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":48311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5953","reported_seq":6003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:22.993478+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:22.993478+0000","last_peered":"2024-09-11T04:59:22.993478+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:22.993478+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731296,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5924","reported_seq":6002,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.500019+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:24.500019+0000","last_peered":"2024-09-11T04:59:24.500019+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:24.500019+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5924,"log_dups_size":0,"ondisk_log_size":5924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48501920,"num_objects":5921,"num_object_clones":0,"num_object_copies":23684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5922,"num_write_kb":47373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3603}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6123","reported_seq":6186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.998957+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:26.998957+0000","last_peered":"2024-09-11T04:59:26.998957+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:26.998957+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":785268928,"num_objects":95863,"num_object_clones":0,"num_object_copies":383452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95890,"num_write_kb":767002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1426866176,"data_stored":1426866176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95917,"ondisk_log_size":95917,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724644,"kb_used_data":381972,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93647196,"statfs":{"total":96636764160,"available":95894728704,"internally_reserved":0,"allocated":391139328,"data_stored":390968518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,3,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720904,"kb_used_data":382904,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93650936,"statfs":{"total":96636764160,"available":95898558464,"internally_reserved":0,"allocated":392093696,"data_stored":391922886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480096,"kb_used_data":241616,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93891744,"statfs":{"total":96636764160,"available":96145145856,"internally_reserved":0,"allocated":247414784,"data_stored":247243974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244182738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724064,"kb_used_data":385616,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93647776,"statfs":{"total":96636764160,"available":95895322624,"internally_reserved":0,"allocated":394870784,"data_stored":394699974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395145216,"data_stored":395145216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248164352,"data_stored":248164352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392482816,"data_stored":392482816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391073792,"data_stored":391073792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:27.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:28.584 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:28.585 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:28.606 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-11T04:59:27.692514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":785268928,"num_objects":95863,"num_object_clones":0,"num_object_copies":383452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95890,"num_write_kb":767002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95917,"ondisk_log_size":95917,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2649708,"kb_used_data":1392108,"kb_used_omap":14,"kb_used_meta":1257521,"kb_avail":374837652,"statfs":{"total":386547056640,"available":383833755648,"internally_reserved":0,"allocated":1425518592,"data_stored":1424835352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1287702070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,5,6,16,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25370624,"num_objects":3097,"num_object_clones":0,"num_object_copies":12388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":24776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1081344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005019"},"pg_stats":[{"pgid":"1.f","version":"31'5926","reported_seq":5968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.305868+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:23.305868+0000","last_peered":"2024-09-11T04:59:23.305868+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:23.305868+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48545792,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":47408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6112","reported_seq":6174,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.973110+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:26.973110+0000","last_peered":"2024-09-11T04:59:26.973110+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:26.973110+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50033824,"num_objects":6108,"num_object_clones":0,"num_object_copies":24432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6110,"num_write_kb":48876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5794","reported_seq":5869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.320778+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:24.320778+0000","last_peered":"2024-09-11T04:59:24.320778+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:24.320778+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47438416,"num_objects":5791,"num_object_clones":0,"num_object_copies":23164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":46334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3571}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5938","reported_seq":6002,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:27.024339+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:27.024339+0000","last_peered":"2024-09-11T04:59:27.024339+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:27.024339+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48608416,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3656}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5952","reported_seq":6006,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.252877+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:23.252877+0000","last_peered":"2024-09-11T04:59:23.252877+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:23.252877+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731296,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":47597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5995","reported_seq":6035,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.305656+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:23.305656+0000","last_peered":"2024-09-11T04:59:23.305656+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:23.305656+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5995,"log_dups_size":0,"ondisk_log_size":5995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49111040,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6117","reported_seq":6180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.873699+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:26.873699+0000","last_peered":"2024-09-11T04:59:26.873699+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:26.873699+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6117,"log_dups_size":0,"ondisk_log_size":6117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058400,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6115,"num_write_kb":48900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6139","reported_seq":6201,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.782139+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:26.782139+0000","last_peered":"2024-09-11T04:59:26.782139+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:26.782139+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246816,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":49084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6066","reported_seq":6122,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.311830+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:23.311830+0000","last_peered":"2024-09-11T04:59:23.311830+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:23.311830+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49621312,"num_objects":6058,"num_object_clones":0,"num_object_copies":24232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6063,"num_write_kb":48481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5988","reported_seq":6066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.682259+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:24.682259+0000","last_peered":"2024-09-11T04:59:24.682259+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:24.682259+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49034400,"num_objects":5986,"num_object_clones":0,"num_object_copies":23944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":47886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3668}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5976","reported_seq":6032,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.364717+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:23.364717+0000","last_peered":"2024-09-11T04:59:23.364717+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:23.364717+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5976,"log_dups_size":0,"ondisk_log_size":5976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48910064,"num_objects":5971,"num_object_clones":0,"num_object_copies":23884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5973,"num_write_kb":47772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5874","reported_seq":10003,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:23.301583+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:23.301583+0000","last_peered":"2024-09-11T04:59:23.301583+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:23.301583+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5874,"log_dups_size":0,"ondisk_log_size":5874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48110160,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6040","reported_seq":6114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.630445+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:24.630445+0000","last_peered":"2024-09-11T04:59:24.630445+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:24.630445+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49470032,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":48311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5953","reported_seq":6003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:22.993478+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:22.993478+0000","last_peered":"2024-09-11T04:59:22.993478+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:22.993478+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731296,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5924","reported_seq":6002,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.500019+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:24.500019+0000","last_peered":"2024-09-11T04:59:24.500019+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:24.500019+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5924,"log_dups_size":0,"ondisk_log_size":5924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48501920,"num_objects":5921,"num_object_clones":0,"num_object_copies":23684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5922,"num_write_kb":47373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3603}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6123","reported_seq":6186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.998957+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:26.998957+0000","last_peered":"2024-09-11T04:59:26.998957+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:26.998957+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":785268928,"num_objects":95863,"num_object_clones":0,"num_object_copies":383452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95890,"num_write_kb":767002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":16752640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1426866176,"data_stored":1426866176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95917,"ondisk_log_size":95917,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724644,"kb_used_data":381972,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93647196,"statfs":{"total":96636764160,"available":95894728704,"internally_reserved":0,"allocated":391139328,"data_stored":390968518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,3,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720904,"kb_used_data":382904,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93650936,"statfs":{"total":96636764160,"available":95898558464,"internally_reserved":0,"allocated":392093696,"data_stored":391922886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480096,"kb_used_data":241616,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93891744,"statfs":{"total":96636764160,"available":96145145856,"internally_reserved":0,"allocated":247414784,"data_stored":247243974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244182738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724064,"kb_used_data":385616,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93647776,"statfs":{"total":96636764160,"available":95895322624,"internally_reserved":0,"allocated":394870784,"data_stored":394699974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395145216,"data_stored":395145216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248164352,"data_stored":248164352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392482816,"data_stored":392482816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391073792,"data_stored":391073792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:28.607 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:28.607 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5926", 'reported_seq': 5968, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:23.305868+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:23.305868+0000', 'last_peered': '2024-09-11T04:59:23.305868+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:23.305868+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5926, 'log_dups_size': 0, 'ondisk_log_size': 5926, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48545792, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5926, 'num_write_kb': 47408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3614}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'6112", 'reported_seq': 6174, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:26.973110+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:26.973110+0000', 'last_peered': '2024-09-11T04:59:26.973110+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:26.973110+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6112, 'log_dups_size': 0, 'ondisk_log_size': 6112, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50033824, 'num_objects': 6108, 'num_object_clones': 0, 'num_object_copies': 24432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6110, 'num_write_kb': 48876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5794", 'reported_seq': 5869, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:24.320778+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:24.320778+0000', 'last_peered': '2024-09-11T04:59:24.320778+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:24.320778+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5794, 'log_dups_size': 0, 'ondisk_log_size': 5794, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47438416, 'num_objects': 5791, 'num_object_clones': 0, 'num_object_copies': 23164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5793, 'num_write_kb': 46334, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3571}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5938", 'reported_seq': 6002, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:27.024339+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:27.024339+0000', 'last_peered': '2024-09-11T04:59:27.024339+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:27.024339+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5938, 'log_dups_size': 0, 'ondisk_log_size': 5938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48608416, 'num_objects': 5934, 'num_object_clones': 0, 'num_object_copies': 23736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5936, 'num_write_kb': 47477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3656}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5952", 'reported_seq': 6006, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:23.252877+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:23.252877+0000', 'last_peered': '2024-09-11T04:59:23.252877+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:23.252877+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5952, 'log_dups_size': 0, 'ondisk_log_size': 5952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48731296, 'num_objects': 5949, 'num_object_clones': 0, 'num_object_copies': 23796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5950, 'num_write_kb': 47597, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3699}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:28.608 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5995", 'reported_seq': 6035, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:23.305656+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:23.305656+0000', 'last_peered': '2024-09-11T04:59:23.305656+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:23.305656+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5995, 'log_dups_size': 0, 'ondisk_log_size': 5995, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49111040, 'num_objects': 5995, 'num_object_clones': 0, 'num_object_copies': 23980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5995, 'num_write_kb': 47960, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3674}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'6117", 'reported_seq': 6180, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:26.873699+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:26.873699+0000', 'last_peered': '2024-09-11T04:59:26.873699+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:26.873699+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6117, 'log_dups_size': 0, 'ondisk_log_size': 6117, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50058400, 'num_objects': 6111, 'num_object_clones': 0, 'num_object_copies': 24444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6115, 'num_write_kb': 48900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3794}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'6139", 'reported_seq': 6201, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:26.782139+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:59:26.782139+0000', 'last_peered': '2024-09-11T04:59:26.782139+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:26.782139+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6139, 'log_dups_size': 0, 'ondisk_log_size': 6139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50246816, 'num_objects': 6134, 'num_object_clones': 0, 'num_object_copies': 24536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6137, 'num_write_kb': 49084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'6066", 'reported_seq': 6122, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:23.311830+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:23.311830+0000', 'last_peered': '2024-09-11T04:59:23.311830+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:23.311830+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6066, 'log_dups_size': 0, 'ondisk_log_size': 6066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49621312, 'num_objects': 6058, 'num_object_clones': 0, 'num_object_copies': 24232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6063, 'num_write_kb': 48481, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3664}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5988", 'reported_seq': 6066, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:24.682259+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:24.682259+0000', 'last_peered': '2024-09-11T04:59:24.682259+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:24.682259+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5988, 'log_dups_size': 0, 'ondisk_log_size': 5988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49034400, 'num_objects': 5986, 'num_object_clones': 0, 'num_object_copies': 23944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5986, 'num_write_kb': 47886, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3668}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:28.609 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5976", 'reported_seq': 6032, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:23.364717+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:23.364717+0000', 'last_peered': '2024-09-11T04:59:23.364717+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:23.364717+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5976, 'log_dups_size': 0, 'ondisk_log_size': 5976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48910064, 'num_objects': 5971, 'num_object_clones': 0, 'num_object_copies': 23884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5973, 'num_write_kb': 47772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3648}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5874", 'reported_seq': 10003, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:23.301583+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:59:23.301583+0000', 'last_peered': '2024-09-11T04:59:23.301583+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:59:23.301583+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5874, 'log_dups_size': 0, 'ondisk_log_size': 5874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48110160, 'num_objects': 5873, 'num_object_clones': 0, 'num_object_copies': 23492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 190, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5873, 'num_write_kb': 46983, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2047, 'num_bytes_recovered': 16752640, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5683}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'6040", 'reported_seq': 6114, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:24.630445+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:24.630445+0000', 'last_peered': '2024-09-11T04:59:24.630445+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:24.630445+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6040, 'log_dups_size': 0, 'ondisk_log_size': 6040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49470032, 'num_objects': 6039, 'num_object_clones': 0, 'num_object_copies': 24156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6039, 'num_write_kb': 48311, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3707}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5953", 'reported_seq': 6003, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:22.993478+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:22.993478+0000', 'last_peered': '2024-09-11T04:59:22.993478+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:22.993478+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5953, 'log_dups_size': 0, 'ondisk_log_size': 5953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48731296, 'num_objects': 5949, 'num_object_clones': 0, 'num_object_copies': 23796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5951, 'num_write_kb': 47604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3662}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5924", 'reported_seq': 6002, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:24.500019+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:24.500019+0000', 'last_peered': '2024-09-11T04:59:24.500019+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:24.500019+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5924, 'log_dups_size': 0, 'ondisk_log_size': 5924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48501920, 'num_objects': 5921, 'num_object_clones': 0, 'num_object_copies': 23684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5922, 'num_write_kb': 47373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3603}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:28.610 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'6123", 'reported_seq': 6186, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:26.998957+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:26.998957+0000', 'last_peered': '2024-09-11T04:59:26.998957+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:26.998957+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6123, 'log_dups_size': 0, 'ondisk_log_size': 6123, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50115744, 'num_objects': 6118, 'num_object_clones': 0, 'num_object_copies': 24472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6121, 'num_write_kb': 48956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3807}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:28.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:28.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 225 20 96897 96877 3.3633 1.95312 0.0449854 0.0464441 2024-09-11T04:59:29.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:29.354 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34837/383452 objects degraded (9.085%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":296,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95863,"data_bytes":785268928,"bytes_used":2713300992,"bytes_avail":383833755648,"bytes_total":386547056640,"degraded_objects":34837,"degraded_total":383452,"degraded_ratio":0.090851006123321823,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":90074,"recovering_keys_per_sec":0,"num_objects_recovered":132,"num_bytes_recovered":1081344,"num_keys_recovered":0,"write_bytes_sec":2113334,"read_op_per_sec":0,"write_op_per_sec":257},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:29.387 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:29.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:29.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 226 20 97134 97114 3.35661 1.85156 0.12462 0.0465336 2024-09-11T04:59:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:29.970 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:30.000 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-11T04:59:29.693065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":790290624,"num_objects":96476,"num_object_clones":0,"num_object_copies":385904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96503,"num_write_kb":771906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":17309696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96530,"ondisk_log_size":96530,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2653364,"kb_used_data":1404084,"kb_used_omap":14,"kb_used_meta":1249201,"kb_avail":374833996,"statfs":{"total":386547056640,"available":383830011904,"internally_reserved":0,"allocated":1437782016,"data_stored":1437098776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1279182390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,7,13,19,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26836992,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1638400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005041"},"pg_stats":[{"pgid":"1.f","version":"31'6019","reported_seq":6060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.463449+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:28.463449+0000","last_peered":"2024-09-11T04:59:28.463449+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:28.463449+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49307648,"num_objects":6019,"num_object_clones":0,"num_object_copies":24076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":48152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3707}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6112","reported_seq":6174,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.973110+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:26.973110+0000","last_peered":"2024-09-11T04:59:26.973110+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:26.973110+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50033824,"num_objects":6108,"num_object_clones":0,"num_object_copies":24432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6110,"num_write_kb":48876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5794","reported_seq":5869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.320778+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:24.320778+0000","last_peered":"2024-09-11T04:59:24.320778+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:24.320778+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47438416,"num_objects":5791,"num_object_clones":0,"num_object_copies":23164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":46334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3571}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5938","reported_seq":6002,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:27.024339+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:27.024339+0000","last_peered":"2024-09-11T04:59:27.024339+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:27.024339+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48608416,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3656}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6034","reported_seq":6088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.223596+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:28.223596+0000","last_peered":"2024-09-11T04:59:28.223596+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:28.223596+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6034,"log_dups_size":0,"ondisk_log_size":6034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49403040,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6032,"num_write_kb":48253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6086","reported_seq":6127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.384407+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:28.384407+0000","last_peered":"2024-09-11T04:59:28.384407+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:28.384407+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49856512,"num_objects":6086,"num_object_clones":0,"num_object_copies":24344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6086,"num_write_kb":48688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6117","reported_seq":6180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.873699+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:26.873699+0000","last_peered":"2024-09-11T04:59:26.873699+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:26.873699+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6117,"log_dups_size":0,"ondisk_log_size":6117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058400,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6115,"num_write_kb":48900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6139","reported_seq":6201,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.782139+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:26.782139+0000","last_peered":"2024-09-11T04:59:26.782139+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:26.782139+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246816,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":49084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6155","reported_seq":6211,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.206620+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:28.206620+0000","last_peered":"2024-09-11T04:59:28.206620+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:28.206620+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50350400,"num_objects":6147,"num_object_clones":0,"num_object_copies":24588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":49193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5988","reported_seq":6066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.682259+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:24.682259+0000","last_peered":"2024-09-11T04:59:24.682259+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:24.682259+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49034400,"num_objects":5986,"num_object_clones":0,"num_object_copies":23944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":47886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3668}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'6070","reported_seq":6126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.415803+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:28.415803+0000","last_peered":"2024-09-11T04:59:28.415803+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:28.415803+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6070,"log_dups_size":0,"ondisk_log_size":6070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49680112,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6067,"num_write_kb":48524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3742}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5958","reported_seq":10222,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.437529+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:28.437529+0000","last_peered":"2024-09-11T04:59:28.437529+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:28.437529+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5958,"log_dups_size":0,"ondisk_log_size":5958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48798288,"num_objects":5957,"num_object_clones":0,"num_object_copies":23828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":47655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2115,"num_bytes_recovered":17309696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5835}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6040","reported_seq":6114,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.630445+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:24.630445+0000","last_peered":"2024-09-11T04:59:24.630445+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:24.630445+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49470032,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":48311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'6033","reported_seq":6082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.433929+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:28.433929+0000","last_peered":"2024-09-11T04:59:28.433929+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:28.433929+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49386656,"num_objects":6029,"num_object_clones":0,"num_object_copies":24116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":48244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3742}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5924","reported_seq":6002,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:24.500019+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:24.500019+0000","last_peered":"2024-09-11T04:59:24.500019+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:24.500019+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5924,"log_dups_size":0,"ondisk_log_size":5924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48501920,"num_objects":5921,"num_object_clones":0,"num_object_copies":23684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5922,"num_write_kb":47373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3603}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6123","reported_seq":6186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.998957+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:26.998957+0000","last_peered":"2024-09-11T04:59:26.998957+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:26.998957+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":790290624,"num_objects":96476,"num_object_clones":0,"num_object_copies":385904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96503,"num_write_kb":771906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":17309696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1438281728,"data_stored":1438281728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96530,"ondisk_log_size":96530,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734144,"kb_used_data":387312,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93637696,"statfs":{"total":96636764160,"available":95885000704,"internally_reserved":0,"allocated":396607488,"data_stored":396436678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720904,"kb_used_data":382904,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93650936,"statfs":{"total":96636764160,"available":95898558464,"internally_reserved":0,"allocated":392093696,"data_stored":391922886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474252,"kb_used_data":248252,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93897588,"statfs":{"total":96636764160,"available":96151130112,"internally_reserved":0,"allocated":254210048,"data_stored":254039238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":231403218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724064,"kb_used_data":385616,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93647776,"statfs":{"total":96636764160,"available":95895322624,"internally_reserved":0,"allocated":394870784,"data_stored":394699974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395145216,"data_stored":395145216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254013440,"data_stored":254013440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392482816,"data_stored":392482816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":396640256,"data_stored":396640256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:30.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 227 20 97396 97376 3.35084 2.04688 0.0751599 0.0466162 2024-09-11T04:59:31.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 228 20 97675 97655 3.3457 2.17969 0.0904999 0.0466907 2024-09-11T04:59:32.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 229 20 97945 97925 3.3403 2.10938 0.0684214 0.0467637 2024-09-11T04:59:33.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:33.643 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:33.644 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:33.676 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-11T04:59:31.693790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":793043136,"num_objects":96812,"num_object_clones":0,"num_object_copies":387248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96839,"num_write_kb":774594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":17309696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96866,"ondisk_log_size":96866,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680988,"kb_used_data":1409500,"kb_used_omap":14,"kb_used_meta":1271409,"kb_avail":374806372,"statfs":{"total":386547056640,"available":383801724928,"internally_reserved":0,"allocated":1443328000,"data_stored":1442644760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1301923382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,11,13,9,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24698880,"num_objects":3015,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":24120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005037"},"pg_stats":[{"pgid":"1.f","version":"31'6019","reported_seq":6060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.463449+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:28.463449+0000","last_peered":"2024-09-11T04:59:28.463449+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:28.463449+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49307648,"num_objects":6019,"num_object_clones":0,"num_object_copies":24076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":48152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3707}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6112","reported_seq":6174,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.973110+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:26.973110+0000","last_peered":"2024-09-11T04:59:26.973110+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:26.973110+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50033824,"num_objects":6108,"num_object_clones":0,"num_object_copies":24432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6110,"num_write_kb":48876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5873","reported_seq":5948,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:29.735879+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:29.735879+0000","last_peered":"2024-09-11T04:59:29.735879+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:29.735879+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5873,"log_dups_size":0,"ondisk_log_size":5873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48085584,"num_objects":5870,"num_object_clones":0,"num_object_copies":23480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":46966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5938","reported_seq":6002,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:27.024339+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:27.024339+0000","last_peered":"2024-09-11T04:59:27.024339+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:27.024339+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48608416,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3656}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6034","reported_seq":6088,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.223596+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:28.223596+0000","last_peered":"2024-09-11T04:59:28.223596+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:28.223596+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6034,"log_dups_size":0,"ondisk_log_size":6034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49403040,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6032,"num_write_kb":48253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6086","reported_seq":6127,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.384407+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:28.384407+0000","last_peered":"2024-09-11T04:59:28.384407+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:28.384407+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49856512,"num_objects":6086,"num_object_clones":0,"num_object_copies":24344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6086,"num_write_kb":48688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6117","reported_seq":6180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.873699+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:26.873699+0000","last_peered":"2024-09-11T04:59:26.873699+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:26.873699+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6117,"log_dups_size":0,"ondisk_log_size":6117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058400,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6115,"num_write_kb":48900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6139","reported_seq":6201,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.782139+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:26.782139+0000","last_peered":"2024-09-11T04:59:26.782139+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:26.782139+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246816,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":49084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6155","reported_seq":6211,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.206620+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:28.206620+0000","last_peered":"2024-09-11T04:59:28.206620+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:28.206620+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50350400,"num_objects":6147,"num_object_clones":0,"num_object_copies":24588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":49193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'6078","reported_seq":6156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:29.612672+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:29.612672+0000","last_peered":"2024-09-11T04:59:29.612672+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:29.612672+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6078,"log_dups_size":0,"ondisk_log_size":6078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49771680,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6076,"num_write_kb":48606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'6070","reported_seq":6126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.415803+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:28.415803+0000","last_peered":"2024-09-11T04:59:28.415803+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:28.415803+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6070,"log_dups_size":0,"ondisk_log_size":6070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49680112,"num_objects":6065,"num_object_clones":0,"num_object_copies":24260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6067,"num_write_kb":48524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3742}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5958","reported_seq":10222,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.437529+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:28.437529+0000","last_peered":"2024-09-11T04:59:28.437529+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:28.437529+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5958,"log_dups_size":0,"ondisk_log_size":5958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48798288,"num_objects":5957,"num_object_clones":0,"num_object_copies":23828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":47655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2115,"num_bytes_recovered":17309696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5835}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6124","reported_seq":6198,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:29.746560+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:29.746560+0000","last_peered":"2024-09-11T04:59:29.746560+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:29.746560+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50158160,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3791}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'6033","reported_seq":6082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:28.433929+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:28.433929+0000","last_peered":"2024-09-11T04:59:28.433929+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:28.433929+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49386656,"num_objects":6029,"num_object_clones":0,"num_object_copies":24116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":48244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3742}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'6007","reported_seq":6085,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:29.643841+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:29.643841+0000","last_peered":"2024-09-11T04:59:29.643841+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:29.643841+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6007,"log_dups_size":0,"ondisk_log_size":6007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49181856,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":48037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6123","reported_seq":6186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:26.998957+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:26.998957+0000","last_peered":"2024-09-11T04:59:26.998957+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:26.998957+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50115744,"num_objects":6118,"num_object_clones":0,"num_object_copies":24472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":793043136,"num_objects":96812,"num_object_clones":0,"num_object_copies":387248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96839,"num_write_kb":774594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2141,"num_bytes_recovered":17309696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1443733504,"data_stored":1443733504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96866,"ondisk_log_size":96866,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734144,"kb_used_data":387312,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93637696,"statfs":{"total":96636764160,"available":95885000704,"internally_reserved":0,"allocated":396607488,"data_stored":396436678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748528,"kb_used_data":388320,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93623312,"statfs":{"total":96636764160,"available":95870271488,"internally_reserved":0,"allocated":397639680,"data_stored":397468870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474252,"kb_used_data":248252,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93897588,"statfs":{"total":96636764160,"available":96151130112,"internally_reserved":0,"allocated":254210048,"data_stored":254039238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":231403218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724064,"kb_used_data":385616,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93647776,"statfs":{"total":96636764160,"available":95895322624,"internally_reserved":0,"allocated":394870784,"data_stored":394699974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346549967},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395145216,"data_stored":395145216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254013440,"data_stored":254013440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397934592,"data_stored":397934592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":396640256,"data_stored":396640256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:33.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:33.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 230 20 98195 98175 3.33427 1.95312 0.0714351 0.0468457 2024-09-11T04:59:34.323 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:34.324 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:34.349 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-11T04:59:33.694457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":801210560,"num_objects":97809,"num_object_clones":0,"num_object_copies":391236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97836,"num_write_kb":782570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2198,"num_bytes_recovered":17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97863,"ondisk_log_size":97863,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2728788,"kb_used_data":1425364,"kb_used_omap":14,"kb_used_meta":1303345,"kb_avail":374758572,"statfs":{"total":386547056640,"available":383752777728,"internally_reserved":0,"allocated":1459572736,"data_stored":1458889496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1334625846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,14,11,20,19,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30253056,"num_objects":3693,"num_object_clones":0,"num_object_copies":14772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":29544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1613824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004812"},"pg_stats":[{"pgid":"1.f","version":"31'6097","reported_seq":6139,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.420563+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:33.420563+0000","last_peered":"2024-09-11T04:59:33.420563+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:33.420563+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49946624,"num_objects":6097,"num_object_clones":0,"num_object_copies":24388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6097,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6195","reported_seq":6256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.989565+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:31.989565+0000","last_peered":"2024-09-11T04:59:31.989565+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:31.989565+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6195,"log_dups_size":0,"ondisk_log_size":6195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50713760,"num_objects":6191,"num_object_clones":0,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6193,"num_write_kb":49540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5873","reported_seq":5948,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:29.735879+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:29.735879+0000","last_peered":"2024-09-11T04:59:29.735879+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:29.735879+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5873,"log_dups_size":0,"ondisk_log_size":5873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48085584,"num_objects":5870,"num_object_clones":0,"num_object_copies":23480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":46966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'6019","reported_seq":6082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.979482+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:31.979482+0000","last_peered":"2024-09-11T04:59:31.979482+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:31.979482+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49271968,"num_objects":6015,"num_object_clones":0,"num_object_copies":24060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":48125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6115","reported_seq":6168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.487313+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:33.487313+0000","last_peered":"2024-09-11T04:59:33.487313+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:33.487313+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50066592,"num_objects":6112,"num_object_clones":0,"num_object_copies":24448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6113,"num_write_kb":48901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6179","reported_seq":6220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.378351+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:33.378351+0000","last_peered":"2024-09-11T04:59:33.378351+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:33.378351+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50618368,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":49432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6194","reported_seq":6257,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.946790+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:31.946790+0000","last_peered":"2024-09-11T04:59:31.946790+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:31.946790+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6194,"log_dups_size":0,"ondisk_log_size":6194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50689184,"num_objects":6188,"num_object_clones":0,"num_object_copies":24752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6192,"num_write_kb":49516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3871}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6221","reported_seq":6283,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.903463+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:31.903463+0000","last_peered":"2024-09-11T04:59:31.903463+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:31.903463+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50918560,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":49740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6219","reported_seq":6275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.502687+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:33.502687+0000","last_peered":"2024-09-11T04:59:33.502687+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:33.502687+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6219,"log_dups_size":0,"ondisk_log_size":6219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50874688,"num_objects":6211,"num_object_clones":0,"num_object_copies":24844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'6078","reported_seq":6156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:29.612672+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:29.612672+0000","last_peered":"2024-09-11T04:59:29.612672+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:29.612672+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6078,"log_dups_size":0,"ondisk_log_size":6078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49771680,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6076,"num_write_kb":48606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'6169","reported_seq":6224,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.529962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:33.529962+0000","last_peered":"2024-09-11T04:59:33.529962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:33.529962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6169,"log_dups_size":0,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50491120,"num_objects":6164,"num_object_clones":0,"num_object_copies":24656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6166,"num_write_kb":49316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'6050","reported_seq":10427,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.491052+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:33.491052+0000","last_peered":"2024-09-11T04:59:33.491052+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:33.491052+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6050,"log_dups_size":0,"ondisk_log_size":6050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49551952,"num_objects":6049,"num_object_clones":0,"num_object_copies":24196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2172,"num_bytes_recovered":17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5984}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6124","reported_seq":6198,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:29.746560+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:29.746560+0000","last_peered":"2024-09-11T04:59:29.746560+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:29.746560+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50158160,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3791}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'6119","reported_seq":6169,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.478299+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:33.478299+0000","last_peered":"2024-09-11T04:59:33.478299+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:33.478299+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50091168,"num_objects":6115,"num_object_clones":0,"num_object_copies":24460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6117,"num_write_kb":48932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'6007","reported_seq":6085,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:29.643841+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:29.643841+0000","last_peered":"2024-09-11T04:59:29.643841+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:29.643841+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6007,"log_dups_size":0,"ondisk_log_size":6007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49181856,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":48037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6204","reported_seq":6267,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.885035+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:31.885035+0000","last_peered":"2024-09-11T04:59:31.885035+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:31.885035+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50779296,"num_objects":6199,"num_object_clones":0,"num_object_copies":24796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":49604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":801210560,"num_objects":97809,"num_object_clones":0,"num_object_copies":391236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97836,"num_write_kb":782570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2198,"num_bytes_recovered":17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1460080640,"data_stored":1460080640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97863,"ondisk_log_size":97863,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744904,"kb_used_data":392504,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93626936,"statfs":{"total":96636764160,"available":95873982464,"internally_reserved":0,"allocated":401924096,"data_stored":401753286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,8,3,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748528,"kb_used_data":388320,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93623312,"statfs":{"total":96636764160,"available":95870271488,"internally_reserved":0,"allocated":397639680,"data_stored":397468870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483864,"kb_used_data":253704,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93887976,"statfs":{"total":96636764160,"available":96141287424,"internally_reserved":0,"allocated":259792896,"data_stored":259622086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235663058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751492,"kb_used_data":390836,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93620348,"statfs":{"total":96636764160,"available":95867236352,"internally_reserved":0,"allocated":400216064,"data_stored":400045254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,0,10,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":400429056,"data_stored":400429056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259686400,"data_stored":259686400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397934592,"data_stored":397934592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402030592,"data_stored":402030592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:34.350 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:34.350 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'6097", 'reported_seq': 6139, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:33.420563+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:33.420563+0000', 'last_peered': '2024-09-11T04:59:33.420563+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:33.420563+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6097, 'log_dups_size': 0, 'ondisk_log_size': 6097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49946624, 'num_objects': 6097, 'num_object_clones': 0, 'num_object_copies': 24388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6097, 'num_write_kb': 48776, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3785}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:34.350 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:34.350 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'6195", 'reported_seq': 6256, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:31.989565+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:31.989565+0000', 'last_peered': '2024-09-11T04:59:31.989565+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:31.989565+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6195, 'log_dups_size': 0, 'ondisk_log_size': 6195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50713760, 'num_objects': 6191, 'num_object_clones': 0, 'num_object_copies': 24764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6193, 'num_write_kb': 49540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3912}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:34.350 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5873", 'reported_seq': 5948, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:29.735879+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:29.735879+0000', 'last_peered': '2024-09-11T04:59:29.735879+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:29.735879+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 5873, 'log_dups_size': 0, 'ondisk_log_size': 5873, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48085584, 'num_objects': 5870, 'num_object_clones': 0, 'num_object_copies': 23480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5872, 'num_write_kb': 46966, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3650}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'6019", 'reported_seq': 6082, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:31.979482+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:31.979482+0000', 'last_peered': '2024-09-11T04:59:31.979482+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:31.979482+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6019, 'log_dups_size': 0, 'ondisk_log_size': 6019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49271968, 'num_objects': 6015, 'num_object_clones': 0, 'num_object_copies': 24060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6017, 'num_write_kb': 48125, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'6115", 'reported_seq': 6168, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:33.487313+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:33.487313+0000', 'last_peered': '2024-09-11T04:59:33.487313+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:33.487313+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6115, 'log_dups_size': 0, 'ondisk_log_size': 6115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50066592, 'num_objects': 6112, 'num_object_clones': 0, 'num_object_copies': 24448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6113, 'num_write_kb': 48901, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3862}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'6179", 'reported_seq': 6220, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:33.378351+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:33.378351+0000', 'last_peered': '2024-09-11T04:59:33.378351+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:33.378351+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6179, 'log_dups_size': 0, 'ondisk_log_size': 6179, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50618368, 'num_objects': 6179, 'num_object_clones': 0, 'num_object_copies': 24716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6179, 'num_write_kb': 49432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3858}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'6194", 'reported_seq': 6257, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:31.946790+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:31.946790+0000', 'last_peered': '2024-09-11T04:59:31.946790+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:31.946790+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6194, 'log_dups_size': 0, 'ondisk_log_size': 6194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50689184, 'num_objects': 6188, 'num_object_clones': 0, 'num_object_copies': 24752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6192, 'num_write_kb': 49516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3871}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:34.351 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'6221", 'reported_seq': 6283, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:31.903463+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:59:31.903463+0000', 'last_peered': '2024-09-11T04:59:31.903463+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:31.903463+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6221, 'log_dups_size': 0, 'ondisk_log_size': 6221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50918560, 'num_objects': 6216, 'num_object_clones': 0, 'num_object_copies': 24864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6219, 'num_write_kb': 49740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'6219", 'reported_seq': 6275, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:33.502687+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:33.502687+0000', 'last_peered': '2024-09-11T04:59:33.502687+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:33.502687+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6219, 'log_dups_size': 0, 'ondisk_log_size': 6219, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50874688, 'num_objects': 6211, 'num_object_clones': 0, 'num_object_copies': 24844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6216, 'num_write_kb': 49705, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3817}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'6078", 'reported_seq': 6156, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:29.612672+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:29.612672+0000', 'last_peered': '2024-09-11T04:59:29.612672+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:29.612672+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6078, 'log_dups_size': 0, 'ondisk_log_size': 6078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49771680, 'num_objects': 6076, 'num_object_clones': 0, 'num_object_copies': 24304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6076, 'num_write_kb': 48606, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3758}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'6169", 'reported_seq': 6224, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:33.529962+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:33.529962+0000', 'last_peered': '2024-09-11T04:59:33.529962+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:33.529962+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6169, 'log_dups_size': 0, 'ondisk_log_size': 6169, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50491120, 'num_objects': 6164, 'num_object_clones': 0, 'num_object_copies': 24656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6166, 'num_write_kb': 49316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3841}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'6050", 'reported_seq': 10427, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:33.491052+0000', 'last_change': '2024-09-11T04:56:20.540244+0000', 'last_active': '2024-09-11T04:59:33.491052+0000', 'last_peered': '2024-09-11T04:59:33.491052+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:59:33.491052+0000', 'last_undegraded': '2024-09-11T04:56:20.249753+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6050, 'log_dups_size': 0, 'ondisk_log_size': 6050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49551952, 'num_objects': 6049, 'num_object_clones': 0, 'num_object_copies': 24196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 65, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6049, 'num_write_kb': 48391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2172, 'num_bytes_recovered': 17776640, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5984}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:34.352 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'6124", 'reported_seq': 6198, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:29.746560+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:29.746560+0000', 'last_peered': '2024-09-11T04:59:29.746560+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:29.746560+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6124, 'log_dups_size': 0, 'ondisk_log_size': 6124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50158160, 'num_objects': 6123, 'num_object_clones': 0, 'num_object_copies': 24492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6123, 'num_write_kb': 48983, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:34.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:34.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'6119", 'reported_seq': 6169, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:33.478299+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:33.478299+0000', 'last_peered': '2024-09-11T04:59:33.478299+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:33.478299+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6119, 'log_dups_size': 0, 'ondisk_log_size': 6119, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50091168, 'num_objects': 6115, 'num_object_clones': 0, 'num_object_copies': 24460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6117, 'num_write_kb': 48932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:34.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:34.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'6007", 'reported_seq': 6085, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:29.643841+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:29.643841+0000', 'last_peered': '2024-09-11T04:59:29.643841+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:29.643841+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6007, 'log_dups_size': 0, 'ondisk_log_size': 6007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49181856, 'num_objects': 6004, 'num_object_clones': 0, 'num_object_copies': 24016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6005, 'num_write_kb': 48037, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:34.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:34.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'6204", 'reported_seq': 6267, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:31.885035+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:31.885035+0000', 'last_peered': '2024-09-11T04:59:31.885035+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:31.885035+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6204, 'log_dups_size': 0, 'ondisk_log_size': 6204, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50779296, 'num_objects': 6199, 'num_object_clones': 0, 'num_object_copies': 24796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6202, 'num_write_kb': 49604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:34.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:34.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 231 20 98436 98416 3.32799 1.88281 0.119378 0.0469353 2024-09-11T04:59:35.121 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:35.122 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34712/391236 objects degraded (8.872%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":302,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":97809,"data_bytes":801210560,"bytes_used":2794278912,"bytes_avail":383752777728,"bytes_total":386547056640,"degraded_objects":34712,"degraded_total":391236,"degraded_ratio":0.088723941559570182,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":134431,"recovering_keys_per_sec":0,"num_objects_recovered":197,"num_bytes_recovered":1613824,"num_keys_recovered":0,"write_bytes_sec":2520077,"read_op_per_sec":0,"write_op_per_sec":307},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:35.146 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:35.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:35.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 232 20 98678 98658 3.32179 1.89062 0.0260962 0.0470231 2024-09-11T04:59:35.805 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:35.806 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:35.833 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-11T04:59:35.695369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":803856576,"num_objects":98132,"num_object_clones":0,"num_object_copies":392528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98159,"num_write_kb":785154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2198,"num_bytes_recovered":17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98186,"ondisk_log_size":98186,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738128,"kb_used_data":1430544,"kb_used_omap":14,"kb_used_meta":1307505,"kb_avail":374749232,"statfs":{"total":386547056640,"available":383743213568,"internally_reserved":0,"allocated":1464877056,"data_stored":1464193816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1338885686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,15,16,28,25,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24559616,"num_objects":2998,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2998,"num_write_kb":23984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004527"},"pg_stats":[{"pgid":"1.f","version":"31'6097","reported_seq":6139,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.420563+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:33.420563+0000","last_peered":"2024-09-11T04:59:33.420563+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:33.420563+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49946624,"num_objects":6097,"num_object_clones":0,"num_object_copies":24388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6097,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6195","reported_seq":6256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.989565+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:31.989565+0000","last_peered":"2024-09-11T04:59:31.989565+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:31.989565+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6195,"log_dups_size":0,"ondisk_log_size":6195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50713760,"num_objects":6191,"num_object_clones":0,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6193,"num_write_kb":49540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5954","reported_seq":6030,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:34.688465+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:34.688465+0000","last_peered":"2024-09-11T04:59:34.688465+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:34.688465+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48749136,"num_objects":5951,"num_object_clones":0,"num_object_copies":23804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5953,"num_write_kb":47614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'6019","reported_seq":6082,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.979482+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:31.979482+0000","last_peered":"2024-09-11T04:59:31.979482+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:31.979482+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49271968,"num_objects":6015,"num_object_clones":0,"num_object_copies":24060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":48125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6115","reported_seq":6168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.487313+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:33.487313+0000","last_peered":"2024-09-11T04:59:33.487313+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:33.487313+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50066592,"num_objects":6112,"num_object_clones":0,"num_object_copies":24448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6113,"num_write_kb":48901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6179","reported_seq":6220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.378351+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:33.378351+0000","last_peered":"2024-09-11T04:59:33.378351+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:33.378351+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50618368,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":49432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6194","reported_seq":6257,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.946790+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:31.946790+0000","last_peered":"2024-09-11T04:59:31.946790+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:31.946790+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6194,"log_dups_size":0,"ondisk_log_size":6194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50689184,"num_objects":6188,"num_object_clones":0,"num_object_copies":24752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6192,"num_write_kb":49516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3871}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6221","reported_seq":6283,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.903463+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:31.903463+0000","last_peered":"2024-09-11T04:59:31.903463+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:31.903463+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50918560,"num_objects":6216,"num_object_clones":0,"num_object_copies":24864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":49740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6219","reported_seq":6275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.502687+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:33.502687+0000","last_peered":"2024-09-11T04:59:33.502687+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:33.502687+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6219,"log_dups_size":0,"ondisk_log_size":6219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50874688,"num_objects":6211,"num_object_clones":0,"num_object_copies":24844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'6159","reported_seq":6236,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:34.573159+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:34.573159+0000","last_peered":"2024-09-11T04:59:34.573159+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:34.573159+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50435232,"num_objects":6157,"num_object_clones":0,"num_object_copies":24628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":49254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'6169","reported_seq":6224,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.529962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:33.529962+0000","last_peered":"2024-09-11T04:59:33.529962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:33.529962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6169,"log_dups_size":0,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50491120,"num_objects":6164,"num_object_clones":0,"num_object_copies":24656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6166,"num_write_kb":49316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'6050","reported_seq":10427,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.491052+0000","last_change":"2024-09-11T04:56:20.540244+0000","last_active":"2024-09-11T04:59:33.491052+0000","last_peered":"2024-09-11T04:59:33.491052+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:33.491052+0000","last_undegraded":"2024-09-11T04:56:20.249753+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6050,"log_dups_size":0,"ondisk_log_size":6050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49551952,"num_objects":6049,"num_object_clones":0,"num_object_copies":24196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2172,"num_bytes_recovered":17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5984}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6213","reported_seq":6286,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:34.685908+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:34.685908+0000","last_peered":"2024-09-11T04:59:34.685908+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:34.685908+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6213,"log_dups_size":0,"ondisk_log_size":6213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50887248,"num_objects":6212,"num_object_clones":0,"num_object_copies":24848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6212,"num_write_kb":49695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'6119","reported_seq":6169,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:33.478299+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:33.478299+0000","last_peered":"2024-09-11T04:59:33.478299+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:33.478299+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50091168,"num_objects":6115,"num_object_clones":0,"num_object_copies":24460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6117,"num_write_kb":48932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'6079","reported_seq":6157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:34.653437+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:34.653437+0000","last_peered":"2024-09-11T04:59:34.653437+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:34.653437+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49771680,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":48613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3758}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6204","reported_seq":6267,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:31.885035+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:31.885035+0000","last_peered":"2024-09-11T04:59:31.885035+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:31.885035+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50779296,"num_objects":6199,"num_object_clones":0,"num_object_copies":24796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":49604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":803856576,"num_objects":98132,"num_object_clones":0,"num_object_copies":392528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98159,"num_write_kb":785154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2198,"num_bytes_recovered":17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1465466880,"data_stored":1465466880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98186,"ondisk_log_size":98186,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744904,"kb_used_data":392504,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93626936,"statfs":{"total":96636764160,"available":95873982464,"internally_reserved":0,"allocated":401924096,"data_stored":401753286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,8,3,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757868,"kb_used_data":393500,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93613972,"statfs":{"total":96636764160,"available":95860707328,"internally_reserved":0,"allocated":402944000,"data_stored":402773190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,9,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483864,"kb_used_data":253704,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93887976,"statfs":{"total":96636764160,"available":96141287424,"internally_reserved":0,"allocated":259792896,"data_stored":259622086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":235663058},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751492,"kb_used_data":390836,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93620348,"statfs":{"total":96636764160,"available":95867236352,"internally_reserved":0,"allocated":400216064,"data_stored":400045254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,0,10,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":400429056,"data_stored":400429056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259686400,"data_stored":259686400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403320832,"data_stored":403320832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402030592,"data_stored":402030592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:36.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 233 20 98952 98932 3.31672 2.14062 0.0663978 0.0470969 2024-09-11T04:59:37.719 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 234 20 99216 99196 3.31132 2.0625 0.0973791 0.04717 2024-09-11T04:59:38.719 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 235 20 99489 99469 3.3063 2.13281 0.047824 0.0472462 2024-09-11T04:59:38.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:39.460 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:39.490 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-11T04:59:38.993304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":811999424,"num_objects":99126,"num_object_clones":0,"num_object_copies":396504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99153,"num_write_kb":793106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99180,"ondisk_log_size":99180,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766256,"kb_used_data":1446192,"kb_used_omap":14,"kb_used_meta":1319985,"kb_avail":374721104,"statfs":{"total":386547056640,"available":383714410496,"internally_reserved":0,"allocated":1480900608,"data_stored":1480217368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1351665206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,11,13,33,16,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26730496,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3263,"num_write_kb":26104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.300790"},"pg_stats":[{"pgid":"1.f","version":"31'6177","reported_seq":6219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.467727+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:38.467727+0000","last_peered":"2024-09-11T04:59:38.467727+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:38.467727+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50601984,"num_objects":6177,"num_object_clones":0,"num_object_copies":24708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6282","reported_seq":6343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:37.058747+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:37.058747+0000","last_peered":"2024-09-11T04:59:37.058747+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:37.058747+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51426464,"num_objects":6278,"num_object_clones":0,"num_object_copies":25112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":50236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5954","reported_seq":6030,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:34.688465+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:34.688465+0000","last_peered":"2024-09-11T04:59:34.688465+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:34.688465+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48749136,"num_objects":5951,"num_object_clones":0,"num_object_copies":23804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5953,"num_write_kb":47614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'6104","reported_seq":6168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.938933+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:36.938933+0000","last_peered":"2024-09-11T04:59:36.938933+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:36.938933+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6104,"log_dups_size":0,"ondisk_log_size":6104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49968288,"num_objects":6100,"num_object_clones":0,"num_object_copies":24400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":48805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6186","reported_seq":6241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.549722+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:38.549722+0000","last_peered":"2024-09-11T04:59:38.549722+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:38.549722+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50648224,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":49469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6247","reported_seq":6288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.241574+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:38.241574+0000","last_peered":"2024-09-11T04:59:38.241574+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:38.241574+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51175424,"num_objects":6247,"num_object_clones":0,"num_object_copies":24988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6247,"num_write_kb":49976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6264","reported_seq":6328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.706114+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:36.706114+0000","last_peered":"2024-09-11T04:59:36.706114+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:36.706114+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51262624,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":50076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3941}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6312","reported_seq":6372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:37.001813+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:37.001813+0000","last_peered":"2024-09-11T04:59:37.001813+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:37.001813+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51664032,"num_objects":6307,"num_object_clones":0,"num_object_copies":25228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":50468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6319","reported_seq":6375,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.630071+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:38.630071+0000","last_peered":"2024-09-11T04:59:38.630071+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:38.630071+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51693888,"num_objects":6311,"num_object_clones":0,"num_object_copies":25244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":50505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'6159","reported_seq":6236,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:34.573159+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:34.573159+0000","last_peered":"2024-09-11T04:59:34.573159+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:34.573159+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6159,"log_dups_size":0,"ondisk_log_size":6159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50435232,"num_objects":6157,"num_object_clones":0,"num_object_copies":24628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":49254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'6256","reported_seq":6311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.633796+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:38.633796+0000","last_peered":"2024-09-11T04:59:38.633796+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:38.633796+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51203824,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6253,"num_write_kb":50012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'6131","reported_seq":10641,"reported_epoch":31,"state":"active+undersized+remapped","last_fresh":"2024-09-11T04:59:38.541803+0000","last_change":"2024-09-11T04:59:38.526797+0000","last_active":"2024-09-11T04:59:38.541803+0000","last_peered":"2024-09-11T04:59:38.541803+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:38.541803+0000","last_undegraded":"2024-09-11T04:59:38.541803+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50215504,"num_objects":6130,"num_object_clones":0,"num_object_copies":24520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":49039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","1(0)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":6130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6213","reported_seq":6286,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:34.685908+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:34.685908+0000","last_peered":"2024-09-11T04:59:34.685908+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:34.685908+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6213,"log_dups_size":0,"ondisk_log_size":6213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50887248,"num_objects":6212,"num_object_clones":0,"num_object_copies":24848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6212,"num_write_kb":49695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'6218","reported_seq":6268,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.549102+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:38.549102+0000","last_peered":"2024-09-11T04:59:38.549102+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:38.549102+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6218,"log_dups_size":0,"ondisk_log_size":6218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50902176,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3927}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'6079","reported_seq":6157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:34.653437+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:34.653437+0000","last_peered":"2024-09-11T04:59:34.653437+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:34.653437+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49771680,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":48613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3758}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6279","reported_seq":6342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.929568+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:36.929568+0000","last_peered":"2024-09-11T04:59:36.929568+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:36.929568+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51393696,"num_objects":6274,"num_object_clones":0,"num_object_copies":25096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":811999424,"num_objects":99126,"num_object_clones":0,"num_object_copies":396504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99153,"num_write_kb":793106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1481674752,"data_stored":1481674752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99180,"ondisk_log_size":99180,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754328,"kb_used_data":397768,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93617512,"statfs":{"total":96636764160,"available":95864332288,"internally_reserved":0,"allocated":407314432,"data_stored":407143622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,11,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757868,"kb_used_data":393500,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93613972,"statfs":{"total":96636764160,"available":95860707328,"internally_reserved":0,"allocated":402944000,"data_stored":402773190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,9,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493400,"kb_used_data":259080,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93878440,"statfs":{"total":96636764160,"available":96131522560,"internally_reserved":0,"allocated":265297920,"data_stored":265127110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239922898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,0,2,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760660,"kb_used_data":395844,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93611180,"statfs":{"total":96636764160,"available":95857848320,"internally_reserved":0,"allocated":405344256,"data_stored":405173446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,6,2,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405692416,"data_stored":405692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265236480,"data_stored":265236480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403320832,"data_stored":403320832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407425024,"data_stored":407425024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:39.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:39.722 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 236 20 99716 99696 3.29981 1.77344 0.0283544 0.0473019 2024-09-11T04:59:40.093 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:40.094 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:40.113 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-11T04:59:40.013900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":814366912,"num_objects":99415,"num_object_clones":0,"num_object_copies":397660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99442,"num_write_kb":795418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99469,"ondisk_log_size":99469,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775716,"kb_used_data":1451488,"kb_used_omap":14,"kb_used_meta":1324145,"kb_avail":374711644,"statfs":{"total":386547056640,"available":383704723456,"internally_reserved":0,"allocated":1486323712,"data_stored":1485635903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1355925046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,5,11,10,28,10,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21323776,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2603,"num_write_kb":20824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.320109"},"pg_stats":[{"pgid":"1.f","version":"31'6177","reported_seq":6219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.467727+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:38.467727+0000","last_peered":"2024-09-11T04:59:38.467727+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:38.467727+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50601984,"num_objects":6177,"num_object_clones":0,"num_object_copies":24708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6282","reported_seq":6343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:37.058747+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:37.058747+0000","last_peered":"2024-09-11T04:59:37.058747+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:37.058747+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51426464,"num_objects":6278,"num_object_clones":0,"num_object_copies":25112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":50236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'6029","reported_seq":6107,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.628483+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:39.628483+0000","last_peered":"2024-09-11T04:59:39.628483+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:39.628483+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49363536,"num_objects":6026,"num_object_clones":0,"num_object_copies":24104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":48214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'6104","reported_seq":6168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.938933+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:36.938933+0000","last_peered":"2024-09-11T04:59:36.938933+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:36.938933+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6104,"log_dups_size":0,"ondisk_log_size":6104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49968288,"num_objects":6100,"num_object_clones":0,"num_object_copies":24400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":48805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6186","reported_seq":6241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.549722+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:38.549722+0000","last_peered":"2024-09-11T04:59:38.549722+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:38.549722+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50648224,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":49469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6247","reported_seq":6288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.241574+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:38.241574+0000","last_peered":"2024-09-11T04:59:38.241574+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:38.241574+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51175424,"num_objects":6247,"num_object_clones":0,"num_object_copies":24988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6247,"num_write_kb":49976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6264","reported_seq":6328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.706114+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:36.706114+0000","last_peered":"2024-09-11T04:59:36.706114+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:36.706114+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51262624,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":50076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3941}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6312","reported_seq":6372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:37.001813+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:37.001813+0000","last_peered":"2024-09-11T04:59:37.001813+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:37.001813+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51664032,"num_objects":6307,"num_object_clones":0,"num_object_copies":25228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":50468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6319","reported_seq":6375,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.630071+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:38.630071+0000","last_peered":"2024-09-11T04:59:38.630071+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:38.630071+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51693888,"num_objects":6311,"num_object_clones":0,"num_object_copies":25244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":50505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'6226","reported_seq":6306,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.634889+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:39.634889+0000","last_peered":"2024-09-11T04:59:39.634889+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:39.634889+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50984096,"num_objects":6224,"num_object_clones":0,"num_object_copies":24896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":49790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'6256","reported_seq":6311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.633796+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:38.633796+0000","last_peered":"2024-09-11T04:59:38.633796+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:38.633796+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51203824,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6253,"num_write_kb":50012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'6131","reported_seq":10641,"reported_epoch":31,"state":"active+undersized+remapped","last_fresh":"2024-09-11T04:59:38.541803+0000","last_change":"2024-09-11T04:59:38.526797+0000","last_active":"2024-09-11T04:59:38.541803+0000","last_peered":"2024-09-11T04:59:38.541803+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:38.541803+0000","last_undegraded":"2024-09-11T04:59:38.541803+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50215504,"num_objects":6130,"num_object_clones":0,"num_object_copies":24520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":49039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","1(0)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":6130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'6291","reported_seq":6368,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.783639+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:39.783639+0000","last_peered":"2024-09-11T04:59:39.783639+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:39.783639+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51526224,"num_objects":6290,"num_object_clones":0,"num_object_copies":25160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6290,"num_write_kb":50319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'6218","reported_seq":6268,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.549102+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:38.549102+0000","last_peered":"2024-09-11T04:59:38.549102+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:38.549102+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6218,"log_dups_size":0,"ondisk_log_size":6218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50902176,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3927}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'6148","reported_seq":6229,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.845832+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:39.845832+0000","last_peered":"2024-09-11T04:59:39.845832+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:39.845832+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50336928,"num_objects":6145,"num_object_clones":0,"num_object_copies":24580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6146,"num_write_kb":49165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6279","reported_seq":6342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.929568+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:36.929568+0000","last_peered":"2024-09-11T04:59:36.929568+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:36.929568+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51393696,"num_objects":6274,"num_object_clones":0,"num_object_copies":25096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":814366912,"num_objects":99415,"num_object_clones":0,"num_object_copies":397660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99442,"num_write_kb":795418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1486839808,"data_stored":1486839808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99469,"ondisk_log_size":99469,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754328,"kb_used_data":397768,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93617512,"statfs":{"total":96636764160,"available":95864332288,"internally_reserved":0,"allocated":407314432,"data_stored":407143622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,11,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":767328,"kb_used_data":398796,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":93604512,"statfs":{"total":96636764160,"available":95851020288,"internally_reserved":0,"allocated":408367104,"data_stored":408191725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493400,"kb_used_data":259080,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93878440,"statfs":{"total":96636764160,"available":96131522560,"internally_reserved":0,"allocated":265297920,"data_stored":265127110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239922898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,0,2,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760660,"kb_used_data":395844,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93611180,"statfs":{"total":96636764160,"available":95857848320,"internally_reserved":0,"allocated":405344256,"data_stored":405173446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,6,2,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405692416,"data_stored":405692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265236480,"data_stored":265236480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408485888,"data_stored":408485888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407425024,"data_stored":407425024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:40.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:40.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'6177", 'reported_seq': 6219, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:38.467727+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:38.467727+0000', 'last_peered': '2024-09-11T04:59:38.467727+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:38.467727+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6177, 'log_dups_size': 0, 'ondisk_log_size': 6177, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50601984, 'num_objects': 6177, 'num_object_clones': 0, 'num_object_copies': 24708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6177, 'num_write_kb': 49416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3865}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:40.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:40.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'6282", 'reported_seq': 6343, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:37.058747+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:37.058747+0000', 'last_peered': '2024-09-11T04:59:37.058747+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:37.058747+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6282, 'log_dups_size': 0, 'ondisk_log_size': 6282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51426464, 'num_objects': 6278, 'num_object_clones': 0, 'num_object_copies': 25112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6280, 'num_write_kb': 50236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3999}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:40.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:40.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'6029", 'reported_seq': 6107, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:39.628483+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:39.628483+0000', 'last_peered': '2024-09-11T04:59:39.628483+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:39.628483+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6029, 'log_dups_size': 0, 'ondisk_log_size': 6029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49363536, 'num_objects': 6026, 'num_object_clones': 0, 'num_object_copies': 24104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6028, 'num_write_kb': 48214, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3806}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:40.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:40.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'6104", 'reported_seq': 6168, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:36.938933+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:36.938933+0000', 'last_peered': '2024-09-11T04:59:36.938933+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:36.938933+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6104, 'log_dups_size': 0, 'ondisk_log_size': 6104, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49968288, 'num_objects': 6100, 'num_object_clones': 0, 'num_object_copies': 24400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6100, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6102, 'num_write_kb': 48805, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:40.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:40.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'6186", 'reported_seq': 6241, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:38.549722+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:38.549722+0000', 'last_peered': '2024-09-11T04:59:38.549722+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:38.549722+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6186, 'log_dups_size': 0, 'ondisk_log_size': 6186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50648224, 'num_objects': 6183, 'num_object_clones': 0, 'num_object_copies': 24732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6184, 'num_write_kb': 49469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3933}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:40.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:40.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'6247", 'reported_seq': 6288, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:38.241574+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:38.241574+0000', 'last_peered': '2024-09-11T04:59:38.241574+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:38.241574+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6247, 'log_dups_size': 0, 'ondisk_log_size': 6247, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51175424, 'num_objects': 6247, 'num_object_clones': 0, 'num_object_copies': 24988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6247, 'num_write_kb': 49976, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3926}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:40.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:40.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'6264", 'reported_seq': 6328, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:36.706114+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:36.706114+0000', 'last_peered': '2024-09-11T04:59:36.706114+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:36.706114+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6264, 'log_dups_size': 0, 'ondisk_log_size': 6264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51262624, 'num_objects': 6258, 'num_object_clones': 0, 'num_object_copies': 25032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6262, 'num_write_kb': 50076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3941}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:40.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:40.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'6312", 'reported_seq': 6372, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:37.001813+0000', 'last_change': '2024-09-11T04:56:20.532611+0000', 'last_active': '2024-09-11T04:59:37.001813+0000', 'last_peered': '2024-09-11T04:59:37.001813+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:37.001813+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6312, 'log_dups_size': 0, 'ondisk_log_size': 6312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51664032, 'num_objects': 6307, 'num_object_clones': 0, 'num_object_copies': 25228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6310, 'num_write_kb': 50468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3946}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:40.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:40.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'6319", 'reported_seq': 6375, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:38.630071+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:38.630071+0000', 'last_peered': '2024-09-11T04:59:38.630071+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:38.630071+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6319, 'log_dups_size': 0, 'ondisk_log_size': 6319, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51693888, 'num_objects': 6311, 'num_object_clones': 0, 'num_object_copies': 25244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6316, 'num_write_kb': 50505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3917}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'6226", 'reported_seq': 6306, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:39.634889+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:39.634889+0000', 'last_peered': '2024-09-11T04:59:39.634889+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:39.634889+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6226, 'log_dups_size': 0, 'ondisk_log_size': 6226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50984096, 'num_objects': 6224, 'num_object_clones': 0, 'num_object_copies': 24896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6224, 'num_write_kb': 49790, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3906}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'6256", 'reported_seq': 6311, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:38.633796+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:38.633796+0000', 'last_peered': '2024-09-11T04:59:38.633796+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:38.633796+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6256, 'log_dups_size': 0, 'ondisk_log_size': 6256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51203824, 'num_objects': 6251, 'num_object_clones': 0, 'num_object_copies': 25004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6253, 'num_write_kb': 50012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3928}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'6131", 'reported_seq': 10641, 'reported_epoch': 31, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-11T04:59:38.541803+0000', 'last_change': '2024-09-11T04:59:38.526797+0000', 'last_active': '2024-09-11T04:59:38.541803+0000', 'last_peered': '2024-09-11T04:59:38.541803+0000', 'last_clean': '2024-09-11T04:54:59.713249+0000', 'last_became_active': '2024-09-11T04:56:20.519105+0000', 'last_became_peered': '2024-09-11T04:56:20.519105+0000', 'last_unstale': '2024-09-11T04:59:38.541803+0000', 'last_undegraded': '2024-09-11T04:59:38.541803+0000', 'last_fullsized': '2024-09-11T04:56:20.227109+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6131, 'log_dups_size': 0, 'ondisk_log_size': 6131, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50215504, 'num_objects': 6130, 'num_object_clones': 0, 'num_object_copies': 24520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6130, 'num_write_kb': 49039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2237, 'num_bytes_recovered': 18309120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '1(0)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 6130}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'6291", 'reported_seq': 6368, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:39.783639+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:39.783639+0000', 'last_peered': '2024-09-11T04:59:39.783639+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:39.783639+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6291, 'log_dups_size': 0, 'ondisk_log_size': 6291, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51526224, 'num_objects': 6290, 'num_object_clones': 0, 'num_object_copies': 25160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6290, 'num_write_kb': 50319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3958}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:40.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:40.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'6218", 'reported_seq': 6268, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:38.549102+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:38.549102+0000', 'last_peered': '2024-09-11T04:59:38.549102+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:38.549102+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6218, 'log_dups_size': 0, 'ondisk_log_size': 6218, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50902176, 'num_objects': 6214, 'num_object_clones': 0, 'num_object_copies': 24856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6216, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3927}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:40.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:40.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'6148", 'reported_seq': 6229, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:39.845832+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:39.845832+0000', 'last_peered': '2024-09-11T04:59:39.845832+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:39.845832+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6148, 'log_dups_size': 0, 'ondisk_log_size': 6148, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50336928, 'num_objects': 6145, 'num_object_clones': 0, 'num_object_copies': 24580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6146, 'num_write_kb': 49165, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3827}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:40.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:40.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'6279", 'reported_seq': 6342, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:36.929568+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:36.929568+0000', 'last_peered': '2024-09-11T04:59:36.929568+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:36.929568+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6279, 'log_dups_size': 0, 'ondisk_log_size': 6279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51393696, 'num_objects': 6274, 'num_object_clones': 0, 'num_object_copies': 25096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6277, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:40.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:40.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 237 20 99920 99900 3.29261 1.59375 0.0985343 0.0474417 2024-09-11T04:59:40.902 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:40.903 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34647/396504 objects degraded (8.738%), 15 pgs degraded, 16 pgs undersized","count":31},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":99126,"data_bytes":811999424,"bytes_used":2832646144,"bytes_avail":383714410496,"bytes_total":386547056640,"degraded_objects":34647,"degraded_total":396504,"degraded_ratio":0.087381211791053806,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":99903,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"write_bytes_sec":2170035,"read_op_per_sec":0,"write_op_per_sec":264},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:40.935 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:40.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:41.604 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-11T04:59:40.013900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":814366912,"num_objects":99415,"num_object_clones":0,"num_object_copies":397660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99442,"num_write_kb":795418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99469,"ondisk_log_size":99469,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775716,"kb_used_data":1451488,"kb_used_omap":14,"kb_used_meta":1324145,"kb_avail":374711644,"statfs":{"total":386547056640,"available":383704723456,"internally_reserved":0,"allocated":1486323712,"data_stored":1485635903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1355925046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,5,11,10,28,10,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21323776,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2603,"num_write_kb":20824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.320109"},"pg_stats":[{"pgid":"1.f","version":"31'6177","reported_seq":6219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.467727+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:38.467727+0000","last_peered":"2024-09-11T04:59:38.467727+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:38.467727+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50601984,"num_objects":6177,"num_object_clones":0,"num_object_copies":24708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'6282","reported_seq":6343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:37.058747+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:37.058747+0000","last_peered":"2024-09-11T04:59:37.058747+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:37.058747+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51426464,"num_objects":6278,"num_object_clones":0,"num_object_copies":25112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":50236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'6029","reported_seq":6107,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.628483+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:39.628483+0000","last_peered":"2024-09-11T04:59:39.628483+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:39.628483+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49363536,"num_objects":6026,"num_object_clones":0,"num_object_copies":24104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":48214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'6104","reported_seq":6168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.938933+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:36.938933+0000","last_peered":"2024-09-11T04:59:36.938933+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:36.938933+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6104,"log_dups_size":0,"ondisk_log_size":6104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49968288,"num_objects":6100,"num_object_clones":0,"num_object_copies":24400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":48805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6186","reported_seq":6241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.549722+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:38.549722+0000","last_peered":"2024-09-11T04:59:38.549722+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:38.549722+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50648224,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":49469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6247","reported_seq":6288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.241574+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:38.241574+0000","last_peered":"2024-09-11T04:59:38.241574+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:38.241574+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51175424,"num_objects":6247,"num_object_clones":0,"num_object_copies":24988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6247,"num_write_kb":49976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6264","reported_seq":6328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.706114+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:36.706114+0000","last_peered":"2024-09-11T04:59:36.706114+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:36.706114+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51262624,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":50076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3941}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6312","reported_seq":6372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:37.001813+0000","last_change":"2024-09-11T04:56:20.532611+0000","last_active":"2024-09-11T04:59:37.001813+0000","last_peered":"2024-09-11T04:59:37.001813+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:37.001813+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51664032,"num_objects":6307,"num_object_clones":0,"num_object_copies":25228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":50468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6319","reported_seq":6375,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.630071+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:38.630071+0000","last_peered":"2024-09-11T04:59:38.630071+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:38.630071+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51693888,"num_objects":6311,"num_object_clones":0,"num_object_copies":25244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":50505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'6226","reported_seq":6306,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.634889+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:39.634889+0000","last_peered":"2024-09-11T04:59:39.634889+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:39.634889+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50984096,"num_objects":6224,"num_object_clones":0,"num_object_copies":24896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":49790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'6256","reported_seq":6311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.633796+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:38.633796+0000","last_peered":"2024-09-11T04:59:38.633796+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:38.633796+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51203824,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6253,"num_write_kb":50012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'6131","reported_seq":10641,"reported_epoch":31,"state":"active+undersized+remapped","last_fresh":"2024-09-11T04:59:38.541803+0000","last_change":"2024-09-11T04:59:38.526797+0000","last_active":"2024-09-11T04:59:38.541803+0000","last_peered":"2024-09-11T04:59:38.541803+0000","last_clean":"2024-09-11T04:54:59.713249+0000","last_became_active":"2024-09-11T04:56:20.519105+0000","last_became_peered":"2024-09-11T04:56:20.519105+0000","last_unstale":"2024-09-11T04:59:38.541803+0000","last_undegraded":"2024-09-11T04:59:38.541803+0000","last_fullsized":"2024-09-11T04:56:20.227109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50215504,"num_objects":6130,"num_object_clones":0,"num_object_copies":24520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":49039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","1(0)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":6130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'6291","reported_seq":6368,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.783639+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:39.783639+0000","last_peered":"2024-09-11T04:59:39.783639+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:39.783639+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51526224,"num_objects":6290,"num_object_clones":0,"num_object_copies":25160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6290,"num_write_kb":50319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'6218","reported_seq":6268,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.549102+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:38.549102+0000","last_peered":"2024-09-11T04:59:38.549102+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:38.549102+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6218,"log_dups_size":0,"ondisk_log_size":6218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50902176,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3927}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'6148","reported_seq":6229,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.845832+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:39.845832+0000","last_peered":"2024-09-11T04:59:39.845832+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:39.845832+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50336928,"num_objects":6145,"num_object_clones":0,"num_object_copies":24580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6146,"num_write_kb":49165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6279","reported_seq":6342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:36.929568+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:36.929568+0000","last_peered":"2024-09-11T04:59:36.929568+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:36.929568+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51393696,"num_objects":6274,"num_object_clones":0,"num_object_copies":25096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":814366912,"num_objects":99415,"num_object_clones":0,"num_object_copies":397660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99442,"num_write_kb":795418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1486839808,"data_stored":1486839808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99469,"ondisk_log_size":99469,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754328,"kb_used_data":397768,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93617512,"statfs":{"total":96636764160,"available":95864332288,"internally_reserved":0,"allocated":407314432,"data_stored":407143622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,11,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":767328,"kb_used_data":398796,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":93604512,"statfs":{"total":96636764160,"available":95851020288,"internally_reserved":0,"allocated":408367104,"data_stored":408191725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493400,"kb_used_data":259080,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93878440,"statfs":{"total":96636764160,"available":96131522560,"internally_reserved":0,"allocated":265297920,"data_stored":265127110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":239922898},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,0,2,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760660,"kb_used_data":395844,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93611180,"statfs":{"total":96636764160,"available":95857848320,"internally_reserved":0,"allocated":405344256,"data_stored":405173446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,6,2,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405692416,"data_stored":405692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265236480,"data_stored":265236480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408485888,"data_stored":408485888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407425024,"data_stored":407425024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:41.719 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 238 20 100153 100133 3.28642 1.82031 0.0714856 0.0475298 2024-09-11T04:59:42.719 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 239 20 100406 100386 3.28094 1.97656 0.0725564 0.0476096 2024-09-11T04:59:43.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:2024-09-11T04:59:43.720455+0000 min lat: 0.00302071 max lat: 2.45895 avg lat: 0.0476898 2024-09-11T04:59:43.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T04:59:43.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout: 240 3 100649 100646 3.27573 2.03125 0.0437616 0.0476898 2024-09-11T04:59:44.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:44.719 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Total time run: 240.046 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Total writes made: 100649 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Write size: 8192 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Object size: 8192 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Bandwidth (MB/sec): 3.27571 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Stddev Bandwidth: 2.23043 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Max bandwidth (MB/sec): 12.5391 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Min bandwidth (MB/sec): 0 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Average IOPS: 419 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Stddev IOPS: 285.494 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Max IOPS: 1605 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Min IOPS: 0 2024-09-11T04:59:44.720 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Average Latency(s): 0.0476901 2024-09-11T04:59:44.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Stddev Latency(s): 0.0465208 2024-09-11T04:59:44.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Max latency(s): 2.45895 2024-09-11T04:59:44.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Min latency(s): 0.00302071 2024-09-11T04:59:44.732 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Cleaning up (deleting benchmark objects) 2024-09-11T04:59:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:45.126 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:45.157 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-11T04:59:43.698600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":818061504,"num_objects":99866,"num_object_clones":0,"num_object_copies":399464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99893,"num_write_kb":799026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":18620416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99920,"ondisk_log_size":99920,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777216,"kb_used_data":1461308,"kb_used_omap":14,"kb_used_meta":1315825,"kb_avail":374710144,"statfs":{"total":386547056640,"available":383703187456,"internally_reserved":0,"allocated":1496379392,"data_stored":1495673265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1347405366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,9,11,27,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14204928,"num_objects":1734,"num_object_clones":0,"num_object_copies":6936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1734,"num_write_kb":13872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003231"},"pg_stats":[{"pgid":"1.f","version":"31'6177","reported_seq":6219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.467727+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:38.467727+0000","last_peered":"2024-09-11T04:59:38.467727+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:38.467727+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50601984,"num_objects":6177,"num_object_clones":0,"num_object_copies":24708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6349","reported_seq":6415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.108332+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:42.108332+0000","last_peered":"2024-09-11T04:59:42.108332+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:42.108332+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51975328,"num_objects":6345,"num_object_clones":0,"num_object_copies":25380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'6029","reported_seq":6107,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.628483+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:39.628483+0000","last_peered":"2024-09-11T04:59:39.628483+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:39.628483+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49363536,"num_objects":6026,"num_object_clones":0,"num_object_copies":24104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":48214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6179","reported_seq":6247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:41.875349+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:41.875349+0000","last_peered":"2024-09-11T04:59:41.875349+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:41.875349+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50582688,"num_objects":6175,"num_object_clones":0,"num_object_copies":24700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3897}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6186","reported_seq":6241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.549722+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:38.549722+0000","last_peered":"2024-09-11T04:59:38.549722+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:38.549722+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50648224,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":49469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6247","reported_seq":6288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.241574+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:38.241574+0000","last_peered":"2024-09-11T04:59:38.241574+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:38.241574+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51175424,"num_objects":6247,"num_object_clones":0,"num_object_copies":24988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6247,"num_write_kb":49976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3926}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6347","reported_seq":6415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.141185+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:42.141185+0000","last_peered":"2024-09-11T04:59:42.141185+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:42.141185+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51942560,"num_objects":6341,"num_object_clones":0,"num_object_copies":25364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6345,"num_write_kb":50740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6396","reported_seq":6538,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.176387+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:42.176387+0000","last_peered":"2024-09-11T04:59:42.176387+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:42.176387+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52352160,"num_objects":6391,"num_object_clones":0,"num_object_copies":25564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":51140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":311296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'6319","reported_seq":6375,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.630071+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:38.630071+0000","last_peered":"2024-09-11T04:59:38.630071+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:38.630071+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51693888,"num_objects":6311,"num_object_clones":0,"num_object_copies":25244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":50505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'6226","reported_seq":6306,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.634889+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:39.634889+0000","last_peered":"2024-09-11T04:59:39.634889+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:39.634889+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50984096,"num_objects":6224,"num_object_clones":0,"num_object_copies":24896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":49790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'6256","reported_seq":6311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.633796+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:38.633796+0000","last_peered":"2024-09-11T04:59:38.633796+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:38.633796+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51203824,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6253,"num_write_kb":50012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6188","reported_seq":10736,"reported_epoch":33,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T04:59:43.280338+0000","last_change":"2024-09-11T04:59:41.030192+0000","last_active":"2024-09-11T04:59:43.280338+0000","last_peered":"2024-09-11T04:59:43.280338+0000","last_clean":"2024-09-11T04:59:43.280338+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:43.280338+0000","last_undegraded":"2024-09-11T04:59:43.280338+0000","last_fullsized":"2024-09-11T04:59:43.280338+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":1949,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50682448,"num_objects":6187,"num_object_clones":0,"num_object_copies":24748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6187,"num_write_kb":49495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"32'6291","reported_seq":6368,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.783639+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:39.783639+0000","last_peered":"2024-09-11T04:59:39.783639+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:39.783639+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51526224,"num_objects":6290,"num_object_clones":0,"num_object_copies":25160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6290,"num_write_kb":50319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'6218","reported_seq":6268,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:38.549102+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:38.549102+0000","last_peered":"2024-09-11T04:59:38.549102+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:38.549102+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6218,"log_dups_size":0,"ondisk_log_size":6218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50902176,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3927}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'6148","reported_seq":6229,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:39.845832+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:39.845832+0000","last_peered":"2024-09-11T04:59:39.845832+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:39.845832+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50336928,"num_objects":6145,"num_object_clones":0,"num_object_copies":24580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6146,"num_write_kb":49165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6364","reported_seq":6431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.040790+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:42.040790+0000","last_peered":"2024-09-11T04:59:42.040790+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:42.040790+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52090016,"num_objects":6359,"num_object_clones":0,"num_object_copies":25436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6362,"num_write_kb":50884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":818061504,"num_objects":99866,"num_object_clones":0,"num_object_copies":399464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99893,"num_write_kb":799026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":18620416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1496961024,"data_stored":1496961024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99920,"ondisk_log_size":99920,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754328,"kb_used_data":397768,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93617512,"statfs":{"total":96636764160,"available":95864332288,"internally_reserved":0,"allocated":407314432,"data_stored":407143622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,11,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":767328,"kb_used_data":398796,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":93604512,"statfs":{"total":96636764160,"available":95851020288,"internally_reserved":0,"allocated":408367104,"data_stored":408191725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502476,"kb_used_data":263996,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93869364,"statfs":{"total":96636764160,"available":96122228736,"internally_reserved":0,"allocated":270331904,"data_stored":270151935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244182738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753084,"kb_used_data":400748,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93618756,"statfs":{"total":96636764160,"available":95865606144,"internally_reserved":0,"allocated":410365952,"data_stored":410185983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410705920,"data_stored":410705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270344192,"data_stored":270344192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408485888,"data_stored":408485888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407425024,"data_stored":407425024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:45.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:45.794 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:45.795 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:45.831 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-11T04:59:45.699580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":823599320,"num_objects":100543,"num_object_clones":0,"num_object_copies":402172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100588,"num_write_kb":804507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":18620416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100615,"ondisk_log_size":100615,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777732,"kb_used_data":1470148,"kb_used_omap":14,"kb_used_meta":1307505,"kb_avail":374709628,"statfs":{"total":386547056640,"available":383702659072,"internally_reserved":0,"allocated":1505431552,"data_stored":1504711676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1338885686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,4,7,15,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16400408,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2021,"num_write_kb":16089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003321"},"pg_stats":[{"pgid":"1.f","version":"33'6257","reported_seq":6303,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.689189+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:43.689189+0000","last_peered":"2024-09-11T04:59:43.689189+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:43.689189+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6257,"log_dups_size":0,"ondisk_log_size":6257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51257344,"num_objects":6257,"num_object_clones":0,"num_object_copies":25028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3945}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6349","reported_seq":6415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.108332+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:42.108332+0000","last_peered":"2024-09-11T04:59:42.108332+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:42.108332+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51975328,"num_objects":6345,"num_object_clones":0,"num_object_copies":25380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6082","reported_seq":6163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.726524+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:44.726524+0000","last_peered":"2024-09-11T04:59:44.726524+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:44.726524+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49789544,"num_objects":6079,"num_object_clones":0,"num_object_copies":24316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6081,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6179","reported_seq":6247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:41.875349+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:41.875349+0000","last_peered":"2024-09-11T04:59:41.875349+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:41.875349+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50582688,"num_objects":6175,"num_object_clones":0,"num_object_copies":24700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3897}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6267","reported_seq":6326,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.582267+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:43.582267+0000","last_peered":"2024-09-11T04:59:43.582267+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:43.582267+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51311776,"num_objects":6264,"num_object_clones":0,"num_object_copies":25056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4014}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6319","reported_seq":6364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.621883+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:43.621883+0000","last_peered":"2024-09-11T04:59:43.621883+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:43.621883+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51765248,"num_objects":6319,"num_object_clones":0,"num_object_copies":25276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":50552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6347","reported_seq":6415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.141185+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:42.141185+0000","last_peered":"2024-09-11T04:59:42.141185+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:42.141185+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51942560,"num_objects":6341,"num_object_clones":0,"num_object_copies":25364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6345,"num_write_kb":50740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6396","reported_seq":6538,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.176387+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:42.176387+0000","last_peered":"2024-09-11T04:59:42.176387+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:42.176387+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52352160,"num_objects":6391,"num_object_clones":0,"num_object_copies":25564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":51140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":311296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6398","reported_seq":6459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.688547+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:43.688547+0000","last_peered":"2024-09-11T04:59:43.688547+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:43.688547+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52341056,"num_objects":6390,"num_object_clones":0,"num_object_copies":25560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6395,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3996}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6268","reported_seq":6350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.779110+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:44.779110+0000","last_peered":"2024-09-11T04:59:44.779110+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:44.779110+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6268,"log_dups_size":0,"ondisk_log_size":6268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51279008,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3942}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6329","reported_seq":6390,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.663106+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:43.663106+0000","last_peered":"2024-09-11T04:59:43.663106+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:43.663106+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51801840,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6326,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6188","reported_seq":10736,"reported_epoch":33,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T04:59:43.280338+0000","last_change":"2024-09-11T04:59:41.030192+0000","last_active":"2024-09-11T04:59:43.280338+0000","last_peered":"2024-09-11T04:59:43.280338+0000","last_clean":"2024-09-11T04:59:43.280338+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:43.280338+0000","last_undegraded":"2024-09-11T04:59:43.280338+0000","last_fullsized":"2024-09-11T04:59:43.280338+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":1949,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50682448,"num_objects":6187,"num_object_clones":0,"num_object_copies":24748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6187,"num_write_kb":49495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6366","reported_seq":6445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.830040+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:44.830040+0000","last_peered":"2024-09-11T04:59:44.830040+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:44.830040+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52075088,"num_objects":6357,"num_object_clones":0,"num_object_copies":25428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6298","reported_seq":6352,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.676151+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:43.676151+0000","last_peered":"2024-09-11T04:59:43.676151+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:43.676151+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51557536,"num_objects":6294,"num_object_clones":0,"num_object_copies":25176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6208","reported_seq":6291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.811179+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:44.811179+0000","last_peered":"2024-09-11T04:59:44.811179+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:44.811179+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50795680,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6364","reported_seq":6431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.040790+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:42.040790+0000","last_peered":"2024-09-11T04:59:42.040790+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:42.040790+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52090016,"num_objects":6359,"num_object_clones":0,"num_object_copies":25436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6362,"num_write_kb":50884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":823599320,"num_objects":100543,"num_object_clones":0,"num_object_copies":402172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100588,"num_write_kb":804507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":18620416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1505492992,"data_stored":1505492992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100615,"ondisk_log_size":100615,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763252,"kb_used_data":402532,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93608588,"statfs":{"total":96636764160,"available":95855194112,"internally_reserved":0,"allocated":412192768,"data_stored":412012799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758920,"kb_used_data":402872,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93612920,"statfs":{"total":96636764160,"available":95859630080,"internally_reserved":0,"allocated":412540928,"data_stored":412360959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502476,"kb_used_data":263996,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93869364,"statfs":{"total":96636764160,"available":96122228736,"internally_reserved":0,"allocated":270331904,"data_stored":270151935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244182738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753084,"kb_used_data":400748,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93618756,"statfs":{"total":96636764160,"available":95865606144,"internally_reserved":0,"allocated":410365952,"data_stored":410185983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410705920,"data_stored":410705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270344192,"data_stored":270344192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412180480,"data_stored":412180480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":412262400,"data_stored":412262400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:45.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:45.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6257", 'reported_seq': 6303, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:43.689189+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:43.689189+0000', 'last_peered': '2024-09-11T04:59:43.689189+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:43.689189+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6257, 'log_dups_size': 0, 'ondisk_log_size': 6257, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51257344, 'num_objects': 6257, 'num_object_clones': 0, 'num_object_copies': 25028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6257, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3945}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:45.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:45.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6349", 'reported_seq': 6415, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:42.108332+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:42.108332+0000', 'last_peered': '2024-09-11T04:59:42.108332+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:42.108332+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6349, 'log_dups_size': 0, 'ondisk_log_size': 6349, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51975328, 'num_objects': 6345, 'num_object_clones': 0, 'num_object_copies': 25380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6347, 'num_write_kb': 50772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4066}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:45.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:45.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6082", 'reported_seq': 6163, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:44.726524+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:44.726524+0000', 'last_peered': '2024-09-11T04:59:44.726524+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:44.726524+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6082, 'log_dups_size': 0, 'ondisk_log_size': 6082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49789544, 'num_objects': 6079, 'num_object_clones': 0, 'num_object_copies': 24316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6079, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6081, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3859}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:45.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:45.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6179", 'reported_seq': 6247, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:41.875349+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:41.875349+0000', 'last_peered': '2024-09-11T04:59:41.875349+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:41.875349+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6179, 'log_dups_size': 0, 'ondisk_log_size': 6179, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50582688, 'num_objects': 6175, 'num_object_clones': 0, 'num_object_copies': 24700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6175, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6177, 'num_write_kb': 49405, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:45.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:45.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6267", 'reported_seq': 6326, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:43.582267+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:43.582267+0000', 'last_peered': '2024-09-11T04:59:43.582267+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:43.582267+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6267, 'log_dups_size': 0, 'ondisk_log_size': 6267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51311776, 'num_objects': 6264, 'num_object_clones': 0, 'num_object_copies': 25056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6265, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4014}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:45.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:45.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6319", 'reported_seq': 6364, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:43.621883+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:43.621883+0000', 'last_peered': '2024-09-11T04:59:43.621883+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:43.621883+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6319, 'log_dups_size': 0, 'ondisk_log_size': 6319, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51765248, 'num_objects': 6319, 'num_object_clones': 0, 'num_object_copies': 25276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6319, 'num_write_kb': 50552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:45.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:45.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6347", 'reported_seq': 6415, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:42.141185+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:42.141185+0000', 'last_peered': '2024-09-11T04:59:42.141185+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:42.141185+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6347, 'log_dups_size': 0, 'ondisk_log_size': 6347, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51942560, 'num_objects': 6341, 'num_object_clones': 0, 'num_object_copies': 25364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6345, 'num_write_kb': 50740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:45.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:45.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6396", 'reported_seq': 6538, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:42.176387+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T04:59:42.176387+0000', 'last_peered': '2024-09-11T04:59:42.176387+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:42.176387+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6396, 'log_dups_size': 0, 'ondisk_log_size': 6396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52352160, 'num_objects': 6391, 'num_object_clones': 0, 'num_object_copies': 25564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2323, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6394, 'num_write_kb': 51140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 311296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4068}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:45.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:45.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6398", 'reported_seq': 6459, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:43.688547+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:43.688547+0000', 'last_peered': '2024-09-11T04:59:43.688547+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:43.688547+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6398, 'log_dups_size': 0, 'ondisk_log_size': 6398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52341056, 'num_objects': 6390, 'num_object_clones': 0, 'num_object_copies': 25560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2397, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6395, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3996}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:45.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:45.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6268", 'reported_seq': 6350, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:44.779110+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:44.779110+0000', 'last_peered': '2024-09-11T04:59:44.779110+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:44.779110+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6268, 'log_dups_size': 0, 'ondisk_log_size': 6268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51279008, 'num_objects': 6260, 'num_object_clones': 0, 'num_object_copies': 25040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6266, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3942}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:45.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:45.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6329", 'reported_seq': 6390, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:43.663106+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:43.663106+0000', 'last_peered': '2024-09-11T04:59:43.663106+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:43.663106+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6329, 'log_dups_size': 0, 'ondisk_log_size': 6329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51801840, 'num_objects': 6324, 'num_object_clones': 0, 'num_object_copies': 25296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6326, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4001}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:45.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-11T04:59:45.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6188", 'reported_seq': 10736, 'reported_epoch': 33, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-11T04:59:43.280338+0000', 'last_change': '2024-09-11T04:59:41.030192+0000', 'last_active': '2024-09-11T04:59:43.280338+0000', 'last_peered': '2024-09-11T04:59:43.280338+0000', 'last_clean': '2024-09-11T04:59:43.280338+0000', 'last_became_active': '2024-09-11T04:59:40.053452+0000', 'last_became_peered': '2024-09-11T04:59:40.053452+0000', 'last_unstale': '2024-09-11T04:59:43.280338+0000', 'last_undegraded': '2024-09-11T04:59:43.280338+0000', 'last_fullsized': '2024-09-11T04:59:43.280338+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 1949, 'log_size': 6188, 'log_dups_size': 0, 'ondisk_log_size': 6188, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 2s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50682448, 'num_objects': 6187, 'num_object_clones': 0, 'num_object_copies': 24748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6187, 'num_write_kb': 49495, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2237, 'num_bytes_recovered': 18309120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [1, 3, 2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-11T04:59:45.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:45.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6366", 'reported_seq': 6445, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:44.830040+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:44.830040+0000', 'last_peered': '2024-09-11T04:59:44.830040+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:44.830040+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6366, 'log_dups_size': 0, 'ondisk_log_size': 6366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52075088, 'num_objects': 6357, 'num_object_clones': 0, 'num_object_copies': 25428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6365, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4025}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:45.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:45.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6298", 'reported_seq': 6352, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:43.676151+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:43.676151+0000', 'last_peered': '2024-09-11T04:59:43.676151+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:43.676151+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6298, 'log_dups_size': 0, 'ondisk_log_size': 6298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51557536, 'num_objects': 6294, 'num_object_clones': 0, 'num_object_copies': 25176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6296, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:45.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:45.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6208", 'reported_seq': 6291, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:44.811179+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:44.811179+0000', 'last_peered': '2024-09-11T04:59:44.811179+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:44.811179+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6208, 'log_dups_size': 0, 'ondisk_log_size': 6208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50795680, 'num_objects': 6201, 'num_object_clones': 0, 'num_object_copies': 24804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6206, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:45.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:45.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6364", 'reported_seq': 6431, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:42.040790+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:42.040790+0000', 'last_peered': '2024-09-11T04:59:42.040790+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:42.040790+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6364, 'log_dups_size': 0, 'ondisk_log_size': 6364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52090016, 'num_objects': 6359, 'num_object_clones': 0, 'num_object_copies': 25436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6362, 'num_write_kb': 50884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4048}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:45.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:46.620 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34607/402172 objects degraded (8.605%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":313,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":100543,"data_bytes":823599320,"bytes_used":2844397568,"bytes_avail":383702659072,"bytes_total":386547056640,"degraded_objects":34607,"degraded_total":402172,"degraded_ratio":0.086050247157932441,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105428,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"read_bytes_sec":127,"write_bytes_sec":2058537,"read_op_per_sec":0,"write_op_per_sec":252},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:46.651 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T04:59:46.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:47.314 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:47.314 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:47.336 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-11T04:59:45.699580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":823599320,"num_objects":100543,"num_object_clones":0,"num_object_copies":402172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100588,"num_write_kb":804507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":18620416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100615,"ondisk_log_size":100615,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777732,"kb_used_data":1470148,"kb_used_omap":14,"kb_used_meta":1307505,"kb_avail":374709628,"statfs":{"total":386547056640,"available":383702659072,"internally_reserved":0,"allocated":1505431552,"data_stored":1504711676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1338885686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,4,7,15,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16400408,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2021,"num_write_kb":16089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003321"},"pg_stats":[{"pgid":"1.f","version":"33'6257","reported_seq":6303,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.689189+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:43.689189+0000","last_peered":"2024-09-11T04:59:43.689189+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:43.689189+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6257,"log_dups_size":0,"ondisk_log_size":6257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51257344,"num_objects":6257,"num_object_clones":0,"num_object_copies":25028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3945}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6349","reported_seq":6415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.108332+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:42.108332+0000","last_peered":"2024-09-11T04:59:42.108332+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:42.108332+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51975328,"num_objects":6345,"num_object_clones":0,"num_object_copies":25380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6082","reported_seq":6163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.726524+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:44.726524+0000","last_peered":"2024-09-11T04:59:44.726524+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:44.726524+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49789544,"num_objects":6079,"num_object_clones":0,"num_object_copies":24316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6081,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6179","reported_seq":6247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:41.875349+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:41.875349+0000","last_peered":"2024-09-11T04:59:41.875349+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:41.875349+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50582688,"num_objects":6175,"num_object_clones":0,"num_object_copies":24700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":49405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3897}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6267","reported_seq":6326,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.582267+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:43.582267+0000","last_peered":"2024-09-11T04:59:43.582267+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:43.582267+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51311776,"num_objects":6264,"num_object_clones":0,"num_object_copies":25056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4014}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6319","reported_seq":6364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.621883+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:43.621883+0000","last_peered":"2024-09-11T04:59:43.621883+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:43.621883+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51765248,"num_objects":6319,"num_object_clones":0,"num_object_copies":25276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":50552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6347","reported_seq":6415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.141185+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:42.141185+0000","last_peered":"2024-09-11T04:59:42.141185+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:42.141185+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51942560,"num_objects":6341,"num_object_clones":0,"num_object_copies":25364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6345,"num_write_kb":50740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6396","reported_seq":6538,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.176387+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:42.176387+0000","last_peered":"2024-09-11T04:59:42.176387+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:42.176387+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52352160,"num_objects":6391,"num_object_clones":0,"num_object_copies":25564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":51140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":311296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6398","reported_seq":6459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.688547+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:43.688547+0000","last_peered":"2024-09-11T04:59:43.688547+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:43.688547+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52341056,"num_objects":6390,"num_object_clones":0,"num_object_copies":25560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6395,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3996}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6268","reported_seq":6350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.779110+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:44.779110+0000","last_peered":"2024-09-11T04:59:44.779110+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:44.779110+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6268,"log_dups_size":0,"ondisk_log_size":6268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51279008,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3942}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6329","reported_seq":6390,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.663106+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:43.663106+0000","last_peered":"2024-09-11T04:59:43.663106+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:43.663106+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51801840,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6326,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6188","reported_seq":10736,"reported_epoch":33,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T04:59:43.280338+0000","last_change":"2024-09-11T04:59:41.030192+0000","last_active":"2024-09-11T04:59:43.280338+0000","last_peered":"2024-09-11T04:59:43.280338+0000","last_clean":"2024-09-11T04:59:43.280338+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:43.280338+0000","last_undegraded":"2024-09-11T04:59:43.280338+0000","last_fullsized":"2024-09-11T04:59:43.280338+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":1949,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50682448,"num_objects":6187,"num_object_clones":0,"num_object_copies":24748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6187,"num_write_kb":49495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":18309120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6366","reported_seq":6445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.830040+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:44.830040+0000","last_peered":"2024-09-11T04:59:44.830040+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:44.830040+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52075088,"num_objects":6357,"num_object_clones":0,"num_object_copies":25428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6298","reported_seq":6352,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:43.676151+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:43.676151+0000","last_peered":"2024-09-11T04:59:43.676151+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:43.676151+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51557536,"num_objects":6294,"num_object_clones":0,"num_object_copies":25176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6208","reported_seq":6291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.811179+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:44.811179+0000","last_peered":"2024-09-11T04:59:44.811179+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:44.811179+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50795680,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6364","reported_seq":6431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:42.040790+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:42.040790+0000","last_peered":"2024-09-11T04:59:42.040790+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:42.040790+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52090016,"num_objects":6359,"num_object_clones":0,"num_object_copies":25436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6362,"num_write_kb":50884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":823599320,"num_objects":100543,"num_object_clones":0,"num_object_copies":402172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100588,"num_write_kb":804507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2301,"num_bytes_recovered":18620416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1505492992,"data_stored":1505492992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100615,"ondisk_log_size":100615,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763252,"kb_used_data":402532,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93608588,"statfs":{"total":96636764160,"available":95855194112,"internally_reserved":0,"allocated":412192768,"data_stored":412012799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758920,"kb_used_data":402872,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93612920,"statfs":{"total":96636764160,"available":95859630080,"internally_reserved":0,"allocated":412540928,"data_stored":412360959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502476,"kb_used_data":263996,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93869364,"statfs":{"total":96636764160,"available":96122228736,"internally_reserved":0,"allocated":270331904,"data_stored":270151935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":244182738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753084,"kb_used_data":400748,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93618756,"statfs":{"total":96636764160,"available":95865606144,"internally_reserved":0,"allocated":410365952,"data_stored":410185983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410705920,"data_stored":410705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270344192,"data_stored":270344192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412180480,"data_stored":412180480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":412262400,"data_stored":412262400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:50.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:50.946 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:50.946 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:50.972 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-11T04:59:49.700933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":819455624,"num_objects":100037,"num_object_clones":0,"num_object_copies":400148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95156,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101372,"ondisk_log_size":101372,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771500,"kb_used_data":1468044,"kb_used_omap":14,"kb_used_meta":1303345,"kb_avail":374715860,"statfs":{"total":386547056640,"available":383709040640,"internally_reserved":0,"allocated":1503277056,"data_stored":1502557180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1334625846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7456200,"num_objects":911,"num_object_clones":0,"num_object_copies":3644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":-3997,"num_write_kb":-37102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2063,"num_bytes_recovered":-16900096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003976"},"pg_stats":[{"pgid":"1.f","version":"33'6321","reported_seq":6367,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.549658+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:48.549658+0000","last_peered":"2024-09-11T04:59:48.549658+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:48.549658+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50733056,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6415","reported_seq":6481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.140046+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:47.140046+0000","last_peered":"2024-09-11T04:59:47.140046+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:47.140046+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51795104,"num_objects":6323,"num_object_clones":0,"num_object_copies":25292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6082","reported_seq":6163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.726524+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:44.726524+0000","last_peered":"2024-09-11T04:59:44.726524+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:44.726524+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49789544,"num_objects":6079,"num_object_clones":0,"num_object_copies":24316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6081,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6247","reported_seq":6315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.202217+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:47.202217+0000","last_peered":"2024-09-11T04:59:47.202217+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:47.202217+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533536,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3891}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6323","reported_seq":6382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.658426+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:48.658426+0000","last_peered":"2024-09-11T04:59:48.658426+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:48.658426+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50853024,"num_objects":6208,"num_object_clones":0,"num_object_copies":24832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6388","reported_seq":6435,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.747544+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:48.747544+0000","last_peered":"2024-09-11T04:59:48.747544+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:48.747544+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6388,"log_dups_size":0,"ondisk_log_size":6388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51216384,"num_objects":6252,"num_object_clones":0,"num_object_copies":25008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6388,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6408","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.106788+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:47.106788+0000","last_peered":"2024-09-11T04:59:47.106788+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:47.106788+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51786912,"num_objects":6322,"num_object_clones":0,"num_object_copies":25288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4005}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6452","reported_seq":6864,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.167082+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:47.167082+0000","last_peered":"2024-09-11T04:59:47.167082+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:47.167082+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6452,"log_dups_size":0,"ondisk_log_size":6452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52204704,"num_objects":6373,"num_object_clones":0,"num_object_copies":25492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6450,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6464","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.738534+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:48.738534+0000","last_peered":"2024-09-11T04:59:48.738534+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:48.738534+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6464,"log_dups_size":0,"ondisk_log_size":6464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51800384,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6461,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6268","reported_seq":6350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.779110+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:44.779110+0000","last_peered":"2024-09-11T04:59:44.779110+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:44.779110+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6268,"log_dups_size":0,"ondisk_log_size":6268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51279008,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3942}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6394","reported_seq":6455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.717147+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:48.717147+0000","last_peered":"2024-09-11T04:59:48.717147+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:48.717147+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51269360,"num_objects":6259,"num_object_clones":0,"num_object_copies":25036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6245","reported_seq":10840,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T04:59:48.297271+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T04:59:48.297271+0000","last_peered":"2024-09-11T04:59:48.297271+0000","last_clean":"2024-09-11T04:59:48.297271+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:48.297271+0000","last_undegraded":"2024-09-11T04:59:48.297271+0000","last_fullsized":"2024-09-11T04:59:48.297271+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50290688,"num_objects":6139,"num_object_clones":0,"num_object_copies":24556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6366","reported_seq":6445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.830040+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:44.830040+0000","last_peered":"2024-09-11T04:59:44.830040+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:44.830040+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52075088,"num_objects":6357,"num_object_clones":0,"num_object_copies":25428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6365","reported_seq":6419,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.666435+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:48.666435+0000","last_peered":"2024-09-11T04:59:48.666435+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:48.666435+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51008672,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6208","reported_seq":6291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.811179+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:44.811179+0000","last_peered":"2024-09-11T04:59:44.811179+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:44.811179+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50795680,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6426","reported_seq":6493,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.030468+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:47.030468+0000","last_peered":"2024-09-11T04:59:47.030468+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:47.030468+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52024480,"num_objects":6351,"num_object_clones":0,"num_object_copies":25404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":819455624,"num_objects":100037,"num_object_clones":0,"num_object_copies":400148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95156,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1500794880,"data_stored":1500794880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101372,"ondisk_log_size":101372,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764104,"kb_used_data":399200,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93607736,"statfs":{"total":96636764160,"available":95854321664,"internally_reserved":0,"allocated":408780800,"data_stored":408600831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758920,"kb_used_data":402872,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93612920,"statfs":{"total":96636764160,"available":95859630080,"internally_reserved":0,"allocated":412540928,"data_stored":412360959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490804,"kb_used_data":264796,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93881036,"statfs":{"total":96636764160,"available":96134180864,"internally_reserved":0,"allocated":271151104,"data_stored":270971135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":231403218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757672,"kb_used_data":401176,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93614168,"statfs":{"total":96636764160,"available":95860908032,"internally_reserved":0,"allocated":410804224,"data_stored":410624255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409755648,"data_stored":409755648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270835712,"data_stored":270835712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412180480,"data_stored":412180480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":408023040,"data_stored":408023040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:50.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:51.691 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:51.721 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-11T04:59:49.700933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":819455624,"num_objects":100037,"num_object_clones":0,"num_object_copies":400148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95156,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101372,"ondisk_log_size":101372,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771500,"kb_used_data":1468044,"kb_used_omap":14,"kb_used_meta":1303345,"kb_avail":374715860,"statfs":{"total":386547056640,"available":383709040640,"internally_reserved":0,"allocated":1503277056,"data_stored":1502557180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1334625846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7456200,"num_objects":911,"num_object_clones":0,"num_object_copies":3644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":-3997,"num_write_kb":-37102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2063,"num_bytes_recovered":-16900096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003976"},"pg_stats":[{"pgid":"1.f","version":"33'6321","reported_seq":6367,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.549658+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:48.549658+0000","last_peered":"2024-09-11T04:59:48.549658+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:48.549658+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50733056,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6415","reported_seq":6481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.140046+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:47.140046+0000","last_peered":"2024-09-11T04:59:47.140046+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:47.140046+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51795104,"num_objects":6323,"num_object_clones":0,"num_object_copies":25292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6082","reported_seq":6163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.726524+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:44.726524+0000","last_peered":"2024-09-11T04:59:44.726524+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:44.726524+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49789544,"num_objects":6079,"num_object_clones":0,"num_object_copies":24316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6081,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6247","reported_seq":6315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.202217+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:47.202217+0000","last_peered":"2024-09-11T04:59:47.202217+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:47.202217+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533536,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3891}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6323","reported_seq":6382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.658426+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:48.658426+0000","last_peered":"2024-09-11T04:59:48.658426+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:48.658426+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50853024,"num_objects":6208,"num_object_clones":0,"num_object_copies":24832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6388","reported_seq":6435,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.747544+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:48.747544+0000","last_peered":"2024-09-11T04:59:48.747544+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:48.747544+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6388,"log_dups_size":0,"ondisk_log_size":6388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51216384,"num_objects":6252,"num_object_clones":0,"num_object_copies":25008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6388,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6408","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.106788+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:47.106788+0000","last_peered":"2024-09-11T04:59:47.106788+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:47.106788+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51786912,"num_objects":6322,"num_object_clones":0,"num_object_copies":25288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4005}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6452","reported_seq":6864,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.167082+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:47.167082+0000","last_peered":"2024-09-11T04:59:47.167082+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:47.167082+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6452,"log_dups_size":0,"ondisk_log_size":6452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52204704,"num_objects":6373,"num_object_clones":0,"num_object_copies":25492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6450,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6464","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.738534+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:48.738534+0000","last_peered":"2024-09-11T04:59:48.738534+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:48.738534+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6464,"log_dups_size":0,"ondisk_log_size":6464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51800384,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6461,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6268","reported_seq":6350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.779110+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:44.779110+0000","last_peered":"2024-09-11T04:59:44.779110+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:44.779110+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6268,"log_dups_size":0,"ondisk_log_size":6268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51279008,"num_objects":6260,"num_object_clones":0,"num_object_copies":25040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3942}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6394","reported_seq":6455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.717147+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:48.717147+0000","last_peered":"2024-09-11T04:59:48.717147+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:48.717147+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51269360,"num_objects":6259,"num_object_clones":0,"num_object_copies":25036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6245","reported_seq":10840,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T04:59:48.297271+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T04:59:48.297271+0000","last_peered":"2024-09-11T04:59:48.297271+0000","last_clean":"2024-09-11T04:59:48.297271+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:48.297271+0000","last_undegraded":"2024-09-11T04:59:48.297271+0000","last_fullsized":"2024-09-11T04:59:48.297271+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50290688,"num_objects":6139,"num_object_clones":0,"num_object_copies":24556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6366","reported_seq":6445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.830040+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:44.830040+0000","last_peered":"2024-09-11T04:59:44.830040+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:44.830040+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52075088,"num_objects":6357,"num_object_clones":0,"num_object_copies":25428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6365","reported_seq":6419,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.666435+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:48.666435+0000","last_peered":"2024-09-11T04:59:48.666435+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:48.666435+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51008672,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6208","reported_seq":6291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:44.811179+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:44.811179+0000","last_peered":"2024-09-11T04:59:44.811179+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:44.811179+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50795680,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6426","reported_seq":6493,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.030468+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:47.030468+0000","last_peered":"2024-09-11T04:59:47.030468+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:47.030468+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52024480,"num_objects":6351,"num_object_clones":0,"num_object_copies":25404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":819455624,"num_objects":100037,"num_object_clones":0,"num_object_copies":400148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95156,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1500794880,"data_stored":1500794880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101372,"ondisk_log_size":101372,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764104,"kb_used_data":399200,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93607736,"statfs":{"total":96636764160,"available":95854321664,"internally_reserved":0,"allocated":408780800,"data_stored":408600831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758920,"kb_used_data":402872,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93612920,"statfs":{"total":96636764160,"available":95859630080,"internally_reserved":0,"allocated":412540928,"data_stored":412360959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490804,"kb_used_data":264796,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93881036,"statfs":{"total":96636764160,"available":96134180864,"internally_reserved":0,"allocated":271151104,"data_stored":270971135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":231403218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757672,"kb_used_data":401176,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93614168,"statfs":{"total":96636764160,"available":95860908032,"internally_reserved":0,"allocated":410804224,"data_stored":410624255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409755648,"data_stored":409755648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270835712,"data_stored":270835712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412180480,"data_stored":412180480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":408023040,"data_stored":408023040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:51.722 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:51.723 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6321", 'reported_seq': 6367, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:48.549658+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:48.549658+0000', 'last_peered': '2024-09-11T04:59:48.549658+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:48.549658+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6321, 'log_dups_size': 0, 'ondisk_log_size': 6321, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50733056, 'num_objects': 6193, 'num_object_clones': 0, 'num_object_copies': 24772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6321, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3881}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:51.723 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:51.723 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6415", 'reported_seq': 6481, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:47.140046+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:47.140046+0000', 'last_peered': '2024-09-11T04:59:47.140046+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:47.140046+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6415, 'log_dups_size': 0, 'ondisk_log_size': 6415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51795104, 'num_objects': 6323, 'num_object_clones': 0, 'num_object_copies': 25292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6413, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:51.723 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:51.723 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6082", 'reported_seq': 6163, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:44.726524+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:44.726524+0000', 'last_peered': '2024-09-11T04:59:44.726524+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:44.726524+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6082, 'log_dups_size': 0, 'ondisk_log_size': 6082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49789544, 'num_objects': 6079, 'num_object_clones': 0, 'num_object_copies': 24316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6079, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6081, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3859}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6247", 'reported_seq': 6315, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:47.202217+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:47.202217+0000', 'last_peered': '2024-09-11T04:59:47.202217+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:47.202217+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6247, 'log_dups_size': 0, 'ondisk_log_size': 6247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50533536, 'num_objects': 6169, 'num_object_clones': 0, 'num_object_copies': 24676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6245, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3891}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6323", 'reported_seq': 6382, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:48.658426+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:48.658426+0000', 'last_peered': '2024-09-11T04:59:48.658426+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:48.658426+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6323, 'log_dups_size': 0, 'ondisk_log_size': 6323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50853024, 'num_objects': 6208, 'num_object_clones': 0, 'num_object_copies': 24832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6321, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3958}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6388", 'reported_seq': 6435, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:48.747544+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:48.747544+0000', 'last_peered': '2024-09-11T04:59:48.747544+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:48.747544+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6388, 'log_dups_size': 0, 'ondisk_log_size': 6388, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51216384, 'num_objects': 6252, 'num_object_clones': 0, 'num_object_copies': 25008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6388, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3932}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6408", 'reported_seq': 6476, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:47.106788+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:47.106788+0000', 'last_peered': '2024-09-11T04:59:47.106788+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:47.106788+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6408, 'log_dups_size': 0, 'ondisk_log_size': 6408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51786912, 'num_objects': 6322, 'num_object_clones': 0, 'num_object_copies': 25288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6406, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4005}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:51.724 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6452", 'reported_seq': 6864, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:47.167082+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T04:59:47.167082+0000', 'last_peered': '2024-09-11T04:59:47.167082+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:47.167082+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6452, 'log_dups_size': 0, 'ondisk_log_size': 6452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52204704, 'num_objects': 6373, 'num_object_clones': 0, 'num_object_copies': 25492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2189, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6450, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 174, 'num_bytes_recovered': 1409024, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6464", 'reported_seq': 6527, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:48.738534+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:48.738534+0000', 'last_peered': '2024-09-11T04:59:48.738534+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:48.738534+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6464, 'log_dups_size': 0, 'ondisk_log_size': 6464, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51800384, 'num_objects': 6324, 'num_object_clones': 0, 'num_object_copies': 25296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2396, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6461, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6268", 'reported_seq': 6350, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:44.779110+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:44.779110+0000', 'last_peered': '2024-09-11T04:59:44.779110+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:44.779110+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6268, 'log_dups_size': 0, 'ondisk_log_size': 6268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51279008, 'num_objects': 6260, 'num_object_clones': 0, 'num_object_copies': 25040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6266, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3942}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6394", 'reported_seq': 6455, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:48.717147+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:48.717147+0000', 'last_peered': '2024-09-11T04:59:48.717147+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:48.717147+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6394, 'log_dups_size': 0, 'ondisk_log_size': 6394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51269360, 'num_objects': 6259, 'num_object_clones': 0, 'num_object_copies': 25036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6391, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3936}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6366", 'reported_seq': 6445, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:44.830040+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:44.830040+0000', 'last_peered': '2024-09-11T04:59:44.830040+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:44.830040+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6366, 'log_dups_size': 0, 'ondisk_log_size': 6366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52075088, 'num_objects': 6357, 'num_object_clones': 0, 'num_object_copies': 25428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6365, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4025}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:51.725 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6365", 'reported_seq': 6419, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:48.666435+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:48.666435+0000', 'last_peered': '2024-09-11T04:59:48.666435+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:48.666435+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6365, 'log_dups_size': 0, 'ondisk_log_size': 6365, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51008672, 'num_objects': 6227, 'num_object_clones': 0, 'num_object_copies': 24908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6363, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:51.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:51.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6208", 'reported_seq': 6291, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:44.811179+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:44.811179+0000', 'last_peered': '2024-09-11T04:59:44.811179+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:44.811179+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6208, 'log_dups_size': 0, 'ondisk_log_size': 6208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50795680, 'num_objects': 6201, 'num_object_clones': 0, 'num_object_copies': 24804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6206, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:51.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:51.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6426", 'reported_seq': 6493, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:47.030468+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:47.030468+0000', 'last_peered': '2024-09-11T04:59:47.030468+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:47.030468+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6426, 'log_dups_size': 0, 'ondisk_log_size': 6426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52024480, 'num_objects': 6351, 'num_object_clones': 0, 'num_object_copies': 25404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6424, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4040}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:51.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:52.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:52.512 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34461/398816 objects degraded (8.641%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":319,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":99704,"data_bytes":816727688,"bytes_used":2837413888,"bytes_avail":383709642752,"bytes_total":386547056640,"degraded_objects":34461,"degraded_total":398816,"degraded_ratio":0.086408268474685065,"read_bytes_sec":87,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:52.534 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T04:59:52.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:53.186 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:53.217 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-11T04:59:51.701698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":816727688,"num_objects":99704,"num_object_clones":0,"num_object_copies":398816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99704,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95489,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101705,"ondisk_log_size":101705,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770912,"kb_used_data":1463288,"kb_used_omap":14,"kb_used_meta":1307505,"kb_avail":374716448,"statfs":{"total":386547056640,"available":383709642752,"internally_reserved":0,"allocated":1498406912,"data_stored":1497687036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1338885686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,13,2,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2360776,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":-3953,"num_write_kb":-39414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2053,"num_bytes_recovered":-16900096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.687798"},"pg_stats":[{"pgid":"1.f","version":"33'6321","reported_seq":6367,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.549658+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:48.549658+0000","last_peered":"2024-09-11T04:59:48.549658+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:48.549658+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50733056,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6415","reported_seq":6481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.140046+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:47.140046+0000","last_peered":"2024-09-11T04:59:47.140046+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:47.140046+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51795104,"num_objects":6323,"num_object_clones":0,"num_object_copies":25292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6161","reported_seq":6250,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:49.876649+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:49.876649+0000","last_peered":"2024-09-11T04:59:49.876649+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:49.876649+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49142376,"num_objects":6000,"num_object_clones":0,"num_object_copies":24000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6160,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3784}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6247","reported_seq":6315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.202217+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:47.202217+0000","last_peered":"2024-09-11T04:59:47.202217+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:47.202217+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533536,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3891}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6323","reported_seq":6382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.658426+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:48.658426+0000","last_peered":"2024-09-11T04:59:48.658426+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:48.658426+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50853024,"num_objects":6208,"num_object_clones":0,"num_object_copies":24832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6388","reported_seq":6435,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.747544+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:48.747544+0000","last_peered":"2024-09-11T04:59:48.747544+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:48.747544+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6388,"log_dups_size":0,"ondisk_log_size":6388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51216384,"num_objects":6252,"num_object_clones":0,"num_object_copies":25008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6388,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6408","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.106788+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:47.106788+0000","last_peered":"2024-09-11T04:59:47.106788+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:47.106788+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51786912,"num_objects":6322,"num_object_clones":0,"num_object_copies":25288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4005}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6452","reported_seq":6864,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.167082+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:47.167082+0000","last_peered":"2024-09-11T04:59:47.167082+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:47.167082+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6452,"log_dups_size":0,"ondisk_log_size":6452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52204704,"num_objects":6373,"num_object_clones":0,"num_object_copies":25492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6450,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6464","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.738534+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:48.738534+0000","last_peered":"2024-09-11T04:59:48.738534+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:48.738534+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6464,"log_dups_size":0,"ondisk_log_size":6464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51800384,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6461,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6349","reported_seq":6437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:49.828803+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:49.828803+0000","last_peered":"2024-09-11T04:59:49.828803+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:49.828803+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615456,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3864}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6394","reported_seq":6455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.717147+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:48.717147+0000","last_peered":"2024-09-11T04:59:48.717147+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:48.717147+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51269360,"num_objects":6259,"num_object_clones":0,"num_object_copies":25036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6245","reported_seq":10840,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T04:59:48.297271+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T04:59:48.297271+0000","last_peered":"2024-09-11T04:59:48.297271+0000","last_clean":"2024-09-11T04:59:48.297271+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:48.297271+0000","last_undegraded":"2024-09-11T04:59:48.297271+0000","last_fullsized":"2024-09-11T04:59:48.297271+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50290688,"num_objects":6139,"num_object_clones":0,"num_object_copies":24556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6458","reported_seq":6537,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:49.855838+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:49.855838+0000","last_peered":"2024-09-11T04:59:49.855838+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:49.855838+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51321424,"num_objects":6265,"num_object_clones":0,"num_object_copies":25060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6457,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6365","reported_seq":6419,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:48.666435+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:48.666435+0000","last_peered":"2024-09-11T04:59:48.666435+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:48.666435+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51008672,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6289","reported_seq":6377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:49.790580+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:49.790580+0000","last_peered":"2024-09-11T04:59:49.790580+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:49.790580+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50132128,"num_objects":6120,"num_object_clones":0,"num_object_copies":24480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6426","reported_seq":6493,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:47.030468+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:47.030468+0000","last_peered":"2024-09-11T04:59:47.030468+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:47.030468+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52024480,"num_objects":6351,"num_object_clones":0,"num_object_copies":25404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":816727688,"num_objects":99704,"num_object_clones":0,"num_object_copies":398816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99704,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95489,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1495326720,"data_stored":1495326720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101705,"ondisk_log_size":101705,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764104,"kb_used_data":399200,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93607736,"statfs":{"total":96636764160,"available":95854321664,"internally_reserved":0,"allocated":408780800,"data_stored":408600831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758332,"kb_used_data":398116,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93613508,"statfs":{"total":96636764160,"available":95860232192,"internally_reserved":0,"allocated":407670784,"data_stored":407490815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490804,"kb_used_data":264796,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93881036,"statfs":{"total":96636764160,"available":96134180864,"internally_reserved":0,"allocated":271151104,"data_stored":270971135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":231403218},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757672,"kb_used_data":401176,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93614168,"statfs":{"total":96636764160,"available":95860908032,"internally_reserved":0,"allocated":410804224,"data_stored":410624255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409755648,"data_stored":409755648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270835712,"data_stored":270835712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406712320,"data_stored":406712320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":408023040,"data_stored":408023040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:56.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:56.854 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:56.855 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:56.891 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-11T04:59:55.703248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":805930632,"num_objects":98386,"num_object_clones":0,"num_object_copies":393544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96807,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103023,"ondisk_log_size":103023,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789748,"kb_used_data":1447432,"kb_used_omap":14,"kb_used_meta":1342193,"kb_avail":374697612,"statfs":{"total":386547056640,"available":383690354688,"internally_reserved":0,"allocated":1482170368,"data_stored":1481450492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1374406198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,2,9,4,16,13,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12130872,"num_objects":-1480,"num_object_clones":0,"num_object_copies":-5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":-3086,"num_write_kb":-43022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2044,"num_bytes_recovered":-16834560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004647"},"pg_stats":[{"pgid":"1.f","version":"33'6394","reported_seq":6440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.736020+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:53.736020+0000","last_peered":"2024-09-11T04:59:53.736020+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:53.736020+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":6120,"num_object_clones":0,"num_object_copies":24480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6488","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.168352+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:52.168352+0000","last_peered":"2024-09-11T04:59:52.168352+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:52.168352+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6488,"log_dups_size":0,"ondisk_log_size":6488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51197088,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6486,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3971}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6234","reported_seq":6323,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.851874+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:54.851874+0000","last_peered":"2024-09-11T04:59:54.851874+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:54.851874+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6234,"log_dups_size":0,"ondisk_log_size":6234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48544360,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6233,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6329","reported_seq":6397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.213407+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:52.213407+0000","last_peered":"2024-09-11T04:59:52.213407+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:52.213407+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49861792,"num_objects":6087,"num_object_clones":0,"num_object_copies":24348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6327,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6408","reported_seq":6466,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.647750+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:53.647750+0000","last_peered":"2024-09-11T04:59:53.647750+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:53.647750+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50156704,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6477","reported_seq":6524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.820412+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:53.820412+0000","last_peered":"2024-09-11T04:59:53.820412+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:53.820412+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50487296,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6477,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3843}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6491","reported_seq":6558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.257675+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:52.257675+0000","last_peered":"2024-09-11T04:59:52.257675+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:52.257675+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51106976,"num_objects":6239,"num_object_clones":0,"num_object_copies":24956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3922}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6536","reported_seq":7039,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.288672+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:52.288672+0000","last_peered":"2024-09-11T04:59:52.288672+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:52.288672+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51516576,"num_objects":6289,"num_object_clones":0,"num_object_copies":25156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6553","reported_seq":6616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.782135+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:53.782135+0000","last_peered":"2024-09-11T04:59:53.782135+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:53.782135+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51071296,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6432","reported_seq":6520,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.946918+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:54.946918+0000","last_peered":"2024-09-11T04:59:54.946918+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:54.946918+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6432,"log_dups_size":0,"ondisk_log_size":6432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49935520,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6430,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6484","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.802937+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:53.802937+0000","last_peered":"2024-09-11T04:59:53.802937+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:53.802937+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50532080,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6481,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6308","reported_seq":10903,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T04:59:53.368518+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T04:59:53.368518+0000","last_peered":"2024-09-11T04:59:53.368518+0000","last_clean":"2024-09-11T04:59:53.368518+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:53.368518+0000","last_undegraded":"2024-09-11T04:59:53.368518+0000","last_fullsized":"2024-09-11T04:59:53.368518+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49774592,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6561","reported_seq":6640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.820456+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:54.820456+0000","last_peered":"2024-09-11T04:59:54.820456+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:54.820456+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50477648,"num_objects":6162,"num_object_clones":0,"num_object_copies":24648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6445","reported_seq":6499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.643667+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:53.643667+0000","last_peered":"2024-09-11T04:59:53.643667+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:53.643667+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50353312,"num_objects":6147,"num_object_clones":0,"num_object_copies":24588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6376","reported_seq":6467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.893192+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:54.893192+0000","last_peered":"2024-09-11T04:59:54.893192+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:54.893192+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49419424,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6374,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6507","reported_seq":6573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.247193+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:52.247193+0000","last_peered":"2024-09-11T04:59:52.247193+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:52.247193+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6507,"log_dups_size":0,"ondisk_log_size":6507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51360928,"num_objects":6270,"num_object_clones":0,"num_object_copies":25080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6505,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":805930632,"num_objects":98386,"num_object_clones":0,"num_object_copies":393544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96807,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1478946816,"data_stored":1478946816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103023,"ondisk_log_size":103023,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763112,"kb_used_data":394056,"kb_used_omap":4,"kb_used_meta":369019,"kb_avail":93608728,"statfs":{"total":96636764160,"available":95855337472,"internally_reserved":0,"allocated":403513344,"data_stored":403333375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":377876176},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,0,8,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757532,"kb_used_data":393148,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614308,"statfs":{"total":96636764160,"available":95861051392,"internally_reserved":0,"allocated":402583552,"data_stored":402403583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512628,"kb_used_data":264420,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93859212,"statfs":{"total":96636764160,"available":96111833088,"internally_reserved":0,"allocated":270766080,"data_stored":270586111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":254144210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756476,"kb_used_data":395808,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93615364,"statfs":{"total":96636764160,"available":95862132736,"internally_reserved":0,"allocated":405307392,"data_stored":405127423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,6,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":404291584,"data_stored":404291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270450688,"data_stored":270450688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401461248,"data_stored":401461248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402743296,"data_stored":402743296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:56.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:57.527 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:57.527 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:57.551 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-11T04:59:55.703248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":805930632,"num_objects":98386,"num_object_clones":0,"num_object_copies":393544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96807,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103023,"ondisk_log_size":103023,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789748,"kb_used_data":1447432,"kb_used_omap":14,"kb_used_meta":1342193,"kb_avail":374697612,"statfs":{"total":386547056640,"available":383690354688,"internally_reserved":0,"allocated":1482170368,"data_stored":1481450492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1374406198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,2,9,4,16,13,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12130872,"num_objects":-1480,"num_object_clones":0,"num_object_copies":-5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":-3086,"num_write_kb":-43022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2044,"num_bytes_recovered":-16834560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004647"},"pg_stats":[{"pgid":"1.f","version":"33'6394","reported_seq":6440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.736020+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:53.736020+0000","last_peered":"2024-09-11T04:59:53.736020+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:53.736020+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":6120,"num_object_clones":0,"num_object_copies":24480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6488","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.168352+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:52.168352+0000","last_peered":"2024-09-11T04:59:52.168352+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:52.168352+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6488,"log_dups_size":0,"ondisk_log_size":6488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51197088,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6486,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3971}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6234","reported_seq":6323,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.851874+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:54.851874+0000","last_peered":"2024-09-11T04:59:54.851874+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:54.851874+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6234,"log_dups_size":0,"ondisk_log_size":6234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48544360,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6233,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6329","reported_seq":6397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.213407+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:52.213407+0000","last_peered":"2024-09-11T04:59:52.213407+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:52.213407+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49861792,"num_objects":6087,"num_object_clones":0,"num_object_copies":24348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6327,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6408","reported_seq":6466,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.647750+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:53.647750+0000","last_peered":"2024-09-11T04:59:53.647750+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:53.647750+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50156704,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6477","reported_seq":6524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.820412+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:53.820412+0000","last_peered":"2024-09-11T04:59:53.820412+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:53.820412+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50487296,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6477,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3843}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6491","reported_seq":6558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.257675+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:52.257675+0000","last_peered":"2024-09-11T04:59:52.257675+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:52.257675+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51106976,"num_objects":6239,"num_object_clones":0,"num_object_copies":24956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3922}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6536","reported_seq":7039,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.288672+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:52.288672+0000","last_peered":"2024-09-11T04:59:52.288672+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:52.288672+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51516576,"num_objects":6289,"num_object_clones":0,"num_object_copies":25156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6553","reported_seq":6616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.782135+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:53.782135+0000","last_peered":"2024-09-11T04:59:53.782135+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:53.782135+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51071296,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6432","reported_seq":6520,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.946918+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:54.946918+0000","last_peered":"2024-09-11T04:59:54.946918+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:54.946918+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6432,"log_dups_size":0,"ondisk_log_size":6432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49935520,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6430,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6484","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.802937+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:53.802937+0000","last_peered":"2024-09-11T04:59:53.802937+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:53.802937+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50532080,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6481,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6308","reported_seq":10903,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T04:59:53.368518+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T04:59:53.368518+0000","last_peered":"2024-09-11T04:59:53.368518+0000","last_clean":"2024-09-11T04:59:53.368518+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:53.368518+0000","last_undegraded":"2024-09-11T04:59:53.368518+0000","last_fullsized":"2024-09-11T04:59:53.368518+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49774592,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6561","reported_seq":6640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.820456+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:54.820456+0000","last_peered":"2024-09-11T04:59:54.820456+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:54.820456+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50477648,"num_objects":6162,"num_object_clones":0,"num_object_copies":24648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6445","reported_seq":6499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.643667+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:53.643667+0000","last_peered":"2024-09-11T04:59:53.643667+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:53.643667+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50353312,"num_objects":6147,"num_object_clones":0,"num_object_copies":24588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6376","reported_seq":6467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.893192+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:54.893192+0000","last_peered":"2024-09-11T04:59:54.893192+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:54.893192+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49419424,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6374,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6507","reported_seq":6573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:52.247193+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:52.247193+0000","last_peered":"2024-09-11T04:59:52.247193+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:52.247193+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6507,"log_dups_size":0,"ondisk_log_size":6507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51360928,"num_objects":6270,"num_object_clones":0,"num_object_copies":25080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6505,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":805930632,"num_objects":98386,"num_object_clones":0,"num_object_copies":393544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96807,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1478946816,"data_stored":1478946816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103023,"ondisk_log_size":103023,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763112,"kb_used_data":394056,"kb_used_omap":4,"kb_used_meta":369019,"kb_avail":93608728,"statfs":{"total":96636764160,"available":95855337472,"internally_reserved":0,"allocated":403513344,"data_stored":403333375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":377876176},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,0,8,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757532,"kb_used_data":393148,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614308,"statfs":{"total":96636764160,"available":95861051392,"internally_reserved":0,"allocated":402583552,"data_stored":402403583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512628,"kb_used_data":264420,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93859212,"statfs":{"total":96636764160,"available":96111833088,"internally_reserved":0,"allocated":270766080,"data_stored":270586111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":254144210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756476,"kb_used_data":395808,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93615364,"statfs":{"total":96636764160,"available":95862132736,"internally_reserved":0,"allocated":405307392,"data_stored":405127423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,6,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":404291584,"data_stored":404291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270450688,"data_stored":270450688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401461248,"data_stored":401461248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402743296,"data_stored":402743296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:59:57.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T04:59:57.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6394", 'reported_seq': 6440, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:53.736020+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:53.736020+0000', 'last_peered': '2024-09-11T04:59:53.736020+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:53.736020+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6394, 'log_dups_size': 0, 'ondisk_log_size': 6394, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50135040, 'num_objects': 6120, 'num_object_clones': 0, 'num_object_copies': 24480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6394, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3808}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6488", 'reported_seq': 6554, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:52.168352+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:52.168352+0000', 'last_peered': '2024-09-11T04:59:52.168352+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:52.168352+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6488, 'log_dups_size': 0, 'ondisk_log_size': 6488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51197088, 'num_objects': 6250, 'num_object_clones': 0, 'num_object_copies': 25000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6486, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3971}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6234", 'reported_seq': 6323, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:54.851874+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:54.851874+0000', 'last_peered': '2024-09-11T04:59:54.851874+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:54.851874+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6234, 'log_dups_size': 0, 'ondisk_log_size': 6234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48544360, 'num_objects': 5927, 'num_object_clones': 0, 'num_object_copies': 23708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2217, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5927, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6233, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3711}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6329", 'reported_seq': 6397, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:52.213407+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:52.213407+0000', 'last_peered': '2024-09-11T04:59:52.213407+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:52.213407+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6329, 'log_dups_size': 0, 'ondisk_log_size': 6329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49861792, 'num_objects': 6087, 'num_object_clones': 0, 'num_object_copies': 24348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6327, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3809}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6408", 'reported_seq': 6466, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:53.647750+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:53.647750+0000', 'last_peered': '2024-09-11T04:59:53.647750+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:53.647750+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6408, 'log_dups_size': 0, 'ondisk_log_size': 6408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50156704, 'num_objects': 6123, 'num_object_clones': 0, 'num_object_copies': 24492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6406, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3873}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:57.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6477", 'reported_seq': 6524, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:53.820412+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:53.820412+0000', 'last_peered': '2024-09-11T04:59:53.820412+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:53.820412+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6477, 'log_dups_size': 0, 'ondisk_log_size': 6477, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50487296, 'num_objects': 6163, 'num_object_clones': 0, 'num_object_copies': 24652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6477, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3843}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6491", 'reported_seq': 6558, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:52.257675+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:52.257675+0000', 'last_peered': '2024-09-11T04:59:52.257675+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:52.257675+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6491, 'log_dups_size': 0, 'ondisk_log_size': 6491, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51106976, 'num_objects': 6239, 'num_object_clones': 0, 'num_object_copies': 24956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6489, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3922}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6536", 'reported_seq': 7039, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:52.288672+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T04:59:52.288672+0000', 'last_peered': '2024-09-11T04:59:52.288672+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:52.288672+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6536, 'log_dups_size': 0, 'ondisk_log_size': 6536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51516576, 'num_objects': 6289, 'num_object_clones': 0, 'num_object_copies': 25156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6534, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 220, 'num_bytes_recovered': 1785856, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4146}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6553", 'reported_seq': 6616, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:53.782135+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:53.782135+0000', 'last_peered': '2024-09-11T04:59:53.782135+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:53.782135+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6553, 'log_dups_size': 0, 'ondisk_log_size': 6553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51071296, 'num_objects': 6235, 'num_object_clones': 0, 'num_object_copies': 24940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2396, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6550, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3842}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6432", 'reported_seq': 6520, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:54.946918+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:54.946918+0000', 'last_peered': '2024-09-11T04:59:54.946918+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:54.946918+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6432, 'log_dups_size': 0, 'ondisk_log_size': 6432, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49935520, 'num_objects': 6096, 'num_object_clones': 0, 'num_object_copies': 24384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6430, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3781}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T04:59:57.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6484", 'reported_seq': 6545, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:53.802937+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:53.802937+0000', 'last_peered': '2024-09-11T04:59:53.802937+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:53.802937+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6484, 'log_dups_size': 0, 'ondisk_log_size': 6484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50532080, 'num_objects': 6169, 'num_object_clones': 0, 'num_object_copies': 24676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6481, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3846}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:57.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T04:59:57.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6561", 'reported_seq': 6640, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:54.820456+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:54.820456+0000', 'last_peered': '2024-09-11T04:59:54.820456+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:54.820456+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6561, 'log_dups_size': 0, 'ondisk_log_size': 6561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50477648, 'num_objects': 6162, 'num_object_clones': 0, 'num_object_copies': 24648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6560, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3830}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:57.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T04:59:57.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6445", 'reported_seq': 6499, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:53.643667+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:53.643667+0000', 'last_peered': '2024-09-11T04:59:53.643667+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:53.643667+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6445, 'log_dups_size': 0, 'ondisk_log_size': 6445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50353312, 'num_objects': 6147, 'num_object_clones': 0, 'num_object_copies': 24588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6443, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:59:57.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T04:59:57.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6376", 'reported_seq': 6467, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:54.893192+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:54.893192+0000', 'last_peered': '2024-09-11T04:59:54.893192+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:54.893192+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6376, 'log_dups_size': 0, 'ondisk_log_size': 6376, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49419424, 'num_objects': 6033, 'num_object_clones': 0, 'num_object_copies': 24132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6374, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3719}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T04:59:57.555 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T04:59:57.555 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6507", 'reported_seq': 6573, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:52.247193+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:52.247193+0000', 'last_peered': '2024-09-11T04:59:52.247193+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:52.247193+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6507, 'log_dups_size': 0, 'ondisk_log_size': 6507, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51360928, 'num_objects': 6270, 'num_object_clones': 0, 'num_object_copies': 25080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6505, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3959}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T04:59:57.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:59:58.299 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:58.299 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34375/391984 objects degraded (8.769%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":325,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":97996,"data_bytes":802735752,"bytes_used":2855546880,"bytes_avail":383691509760,"bytes_total":386547056640,"degraded_objects":34375,"degraded_total":391984,"degraded_ratio":0.087694905914527124},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T04:59:58.320 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T04:59:58.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:59:58.939 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T04:59:58.941 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T04:59:58.969 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-11T04:59:57.703943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":802735752,"num_objects":97996,"num_object_clones":0,"num_object_copies":391984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97197,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103413,"ondisk_log_size":103413,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788620,"kb_used_data":1442144,"kb_used_omap":14,"kb_used_meta":1346353,"kb_avail":374698740,"statfs":{"total":386547056640,"available":383691509760,"internally_reserved":0,"allocated":1476755456,"data_stored":1476035580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1378666038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,4,8,18,15,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20863568,"num_objects":-2547,"num_object_clones":0,"num_object_copies":-10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3391,"num_write_kb":-48503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2005,"num_bytes_recovered":-16515072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004363"},"pg_stats":[{"pgid":"1.f","version":"33'6394","reported_seq":6440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.736020+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:53.736020+0000","last_peered":"2024-09-11T04:59:53.736020+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:53.736020+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":6120,"num_object_clones":0,"num_object_copies":24480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6557","reported_seq":6624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.364662+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:57.364662+0000","last_peered":"2024-09-11T04:59:57.364662+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:57.364662+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6557,"log_dups_size":0,"ondisk_log_size":6557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50631840,"num_objects":6181,"num_object_clones":0,"num_object_copies":24724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6234","reported_seq":6323,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.851874+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:54.851874+0000","last_peered":"2024-09-11T04:59:54.851874+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:54.851874+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6234,"log_dups_size":0,"ondisk_log_size":6234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48544360,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6233,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6399","reported_seq":6467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.119767+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:57.119767+0000","last_peered":"2024-09-11T04:59:57.119767+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:57.119767+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49288352,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6397,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6408","reported_seq":6466,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.647750+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:53.647750+0000","last_peered":"2024-09-11T04:59:53.647750+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:53.647750+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50156704,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6477","reported_seq":6524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.820412+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:53.820412+0000","last_peered":"2024-09-11T04:59:53.820412+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:53.820412+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50487296,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6477,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3843}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6562","reported_seq":6630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.215818+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:57.215818+0000","last_peered":"2024-09-11T04:59:57.215818+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:57.215818+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525344,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3851}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6624","reported_seq":7204,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.416323+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:57.416323+0000","last_peered":"2024-09-11T04:59:57.416323+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:57.416323+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50795680,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6553","reported_seq":6616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.782135+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:53.782135+0000","last_peered":"2024-09-11T04:59:53.782135+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:53.782135+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51071296,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6432","reported_seq":6520,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.946918+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:54.946918+0000","last_peered":"2024-09-11T04:59:54.946918+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:54.946918+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6432,"log_dups_size":0,"ondisk_log_size":6432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49935520,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6430,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6484","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.802937+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:53.802937+0000","last_peered":"2024-09-11T04:59:53.802937+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:53.802937+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50532080,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6481,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6308","reported_seq":10903,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T04:59:53.368518+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T04:59:53.368518+0000","last_peered":"2024-09-11T04:59:53.368518+0000","last_clean":"2024-09-11T04:59:53.368518+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:53.368518+0000","last_undegraded":"2024-09-11T04:59:53.368518+0000","last_fullsized":"2024-09-11T04:59:53.368518+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49774592,"num_objects":6076,"num_object_clones":0,"num_object_copies":24304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6561","reported_seq":6640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.820456+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:54.820456+0000","last_peered":"2024-09-11T04:59:54.820456+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:54.820456+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50477648,"num_objects":6162,"num_object_clones":0,"num_object_copies":24648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6445","reported_seq":6499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:53.643667+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:53.643667+0000","last_peered":"2024-09-11T04:59:53.643667+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:53.643667+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50353312,"num_objects":6147,"num_object_clones":0,"num_object_copies":24588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6376","reported_seq":6467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:54.893192+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:54.893192+0000","last_peered":"2024-09-11T04:59:54.893192+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:54.893192+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49419424,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6374,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6599","reported_seq":6667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.300158+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:57.300158+0000","last_peered":"2024-09-11T04:59:57.300158+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:57.300158+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50607264,"num_objects":6178,"num_object_clones":0,"num_object_copies":24712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6597,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":802735752,"num_objects":97996,"num_object_clones":0,"num_object_copies":391984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97197,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1473552384,"data_stored":1473552384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103413,"ondisk_log_size":103413,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763112,"kb_used_data":394056,"kb_used_omap":4,"kb_used_meta":369019,"kb_avail":93608728,"statfs":{"total":96636764160,"available":95855337472,"internally_reserved":0,"allocated":403513344,"data_stored":403333375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":377876176},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,0,8,3,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757532,"kb_used_data":393148,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614308,"statfs":{"total":96636764160,"available":95861051392,"internally_reserved":0,"allocated":402583552,"data_stored":402403583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512628,"kb_used_data":264420,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93859212,"statfs":{"total":96636764160,"available":96111833088,"internally_reserved":0,"allocated":270766080,"data_stored":270586111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":254144210},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755348,"kb_used_data":390520,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93616492,"statfs":{"total":96636764160,"available":95863287808,"internally_reserved":0,"allocated":399892480,"data_stored":399712511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":398897152,"data_stored":398897152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270450688,"data_stored":270450688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401461248,"data_stored":401461248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402743296,"data_stored":402743296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:01.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:02.646 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-11T05:00:01.705599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":795338376,"num_objects":97093,"num_object_clones":0,"num_object_copies":388372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98100,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104316,"ondisk_log_size":104316,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2791616,"kb_used_data":1431284,"kb_used_omap":14,"kb_used_meta":1360241,"kb_avail":374695744,"statfs":{"total":386547056640,"available":383688441856,"internally_reserved":0,"allocated":1465634816,"data_stored":1464914940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1392887350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,9,14,6,2,0,0,1,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24117248,"num_objects":-2944,"num_object_clones":0,"num_object_copies":-11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":696320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004666"},"pg_stats":[{"pgid":"1.f","version":"33'6481","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.884401+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:58.884401+0000","last_peered":"2024-09-11T04:59:58.884401+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:58.884401+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6481,"log_dups_size":0,"ondisk_log_size":6481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49422336,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6481,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6557","reported_seq":6624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.364662+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:57.364662+0000","last_peered":"2024-09-11T04:59:57.364662+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:57.364662+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6557,"log_dups_size":0,"ondisk_log_size":6557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50631840,"num_objects":6181,"num_object_clones":0,"num_object_copies":24724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6314","reported_seq":6403,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.834518+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:59.834518+0000","last_peered":"2024-09-11T04:59:59.834518+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:59.834518+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47889000,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6313,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3631}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6399","reported_seq":6467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.119767+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:57.119767+0000","last_peered":"2024-09-11T04:59:57.119767+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:57.119767+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49288352,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6397,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6471","reported_seq":6532,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.790732+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:58.790732+0000","last_peered":"2024-09-11T04:59:58.790732+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:58.790732+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49640608,"num_objects":6060,"num_object_clones":0,"num_object_copies":24240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6469,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6555","reported_seq":6602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.836530+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:58.836530+0000","last_peered":"2024-09-11T04:59:58.836530+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:58.836530+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6555,"log_dups_size":0,"ondisk_log_size":6555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49848320,"num_objects":6085,"num_object_clones":0,"num_object_copies":24340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6562","reported_seq":6630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.215818+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:57.215818+0000","last_peered":"2024-09-11T04:59:57.215818+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:57.215818+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525344,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3851}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6624","reported_seq":7204,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.416323+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:57.416323+0000","last_peered":"2024-09-11T04:59:57.416323+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:57.416323+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50795680,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6639","reported_seq":6704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.672728+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:58.672728+0000","last_peered":"2024-09-11T04:59:58.672728+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:58.672728+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50366784,"num_objects":6149,"num_object_clones":0,"num_object_copies":24596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6636,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6520","reported_seq":6607,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.984189+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:59.984189+0000","last_peered":"2024-09-11T04:59:59.984189+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:59.984189+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6520,"log_dups_size":0,"ondisk_log_size":6520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49214624,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6518,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6571","reported_seq":6632,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.827954+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:58.827954+0000","last_peered":"2024-09-11T04:59:58.827954+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:58.827954+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49819376,"num_objects":6082,"num_object_clones":0,"num_object_copies":24328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6406","reported_seq":11001,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T04:59:58.324824+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T04:59:58.324824+0000","last_peered":"2024-09-11T04:59:58.324824+0000","last_clean":"2024-09-11T04:59:58.324824+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:58.324824+0000","last_undegraded":"2024-09-11T04:59:58.324824+0000","last_fullsized":"2024-09-11T04:59:58.324824+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48971776,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6637","reported_seq":6716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.989985+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:59.989985+0000","last_peered":"2024-09-11T04:59:59.989985+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:59.989985+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6637,"log_dups_size":0,"ondisk_log_size":6637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49855056,"num_objects":6086,"num_object_clones":0,"num_object_copies":24344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6636,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6532","reported_seq":6587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.858163+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:58.858163+0000","last_peered":"2024-09-11T04:59:58.858163+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:58.858163+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49640608,"num_objects":6060,"num_object_clones":0,"num_object_copies":24240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6449","reported_seq":6542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.866382+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:59.866382+0000","last_peered":"2024-09-11T04:59:59.866382+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:59.866382+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821408,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6599","reported_seq":6667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.300158+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:57.300158+0000","last_peered":"2024-09-11T04:59:57.300158+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:57.300158+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50607264,"num_objects":6178,"num_object_clones":0,"num_object_copies":24712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6597,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":795338376,"num_objects":97093,"num_object_clones":0,"num_object_copies":388372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98100,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1462468608,"data_stored":1462468608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104316,"ondisk_log_size":104316,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745416,"kb_used_data":388856,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93626424,"statfs":{"total":96636764160,"available":95873458176,"internally_reserved":0,"allocated":398188544,"data_stored":398008575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,6,3,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756616,"kb_used_data":388088,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":93615224,"statfs":{"total":96636764160,"available":95861989376,"internally_reserved":0,"allocated":397402112,"data_stored":397222143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,0,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534236,"kb_used_data":263820,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93837604,"statfs":{"total":96636764160,"available":96089706496,"internally_reserved":0,"allocated":270151680,"data_stored":269971711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276885202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755348,"kb_used_data":390520,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93616492,"statfs":{"total":96636764160,"available":95863287808,"internally_reserved":0,"allocated":399892480,"data_stored":399712511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":398897152,"data_stored":398897152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269864960,"data_stored":269864960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396283904,"data_stored":396283904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397422592,"data_stored":397422592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:02.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:03.310 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-11T05:00:01.705599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":795338376,"num_objects":97093,"num_object_clones":0,"num_object_copies":388372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98100,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104316,"ondisk_log_size":104316,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2791616,"kb_used_data":1431284,"kb_used_omap":14,"kb_used_meta":1360241,"kb_avail":374695744,"statfs":{"total":386547056640,"available":383688441856,"internally_reserved":0,"allocated":1465634816,"data_stored":1464914940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1392887350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,9,14,6,2,0,0,1,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24117248,"num_objects":-2944,"num_object_clones":0,"num_object_copies":-11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":696320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004666"},"pg_stats":[{"pgid":"1.f","version":"33'6481","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.884401+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:58.884401+0000","last_peered":"2024-09-11T04:59:58.884401+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:58.884401+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6481,"log_dups_size":0,"ondisk_log_size":6481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49422336,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6481,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6557","reported_seq":6624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.364662+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T04:59:57.364662+0000","last_peered":"2024-09-11T04:59:57.364662+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T04:59:57.364662+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6557,"log_dups_size":0,"ondisk_log_size":6557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50631840,"num_objects":6181,"num_object_clones":0,"num_object_copies":24724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6314","reported_seq":6403,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.834518+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:59.834518+0000","last_peered":"2024-09-11T04:59:59.834518+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:59.834518+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47889000,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6313,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3631}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6399","reported_seq":6467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.119767+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T04:59:57.119767+0000","last_peered":"2024-09-11T04:59:57.119767+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T04:59:57.119767+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49288352,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6397,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6471","reported_seq":6532,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.790732+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:58.790732+0000","last_peered":"2024-09-11T04:59:58.790732+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:58.790732+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49640608,"num_objects":6060,"num_object_clones":0,"num_object_copies":24240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6469,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6555","reported_seq":6602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.836530+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:58.836530+0000","last_peered":"2024-09-11T04:59:58.836530+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:58.836530+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6555,"log_dups_size":0,"ondisk_log_size":6555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49848320,"num_objects":6085,"num_object_clones":0,"num_object_copies":24340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6562","reported_seq":6630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.215818+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T04:59:57.215818+0000","last_peered":"2024-09-11T04:59:57.215818+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T04:59:57.215818+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525344,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3851}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6624","reported_seq":7204,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.416323+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T04:59:57.416323+0000","last_peered":"2024-09-11T04:59:57.416323+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T04:59:57.416323+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50795680,"num_objects":6201,"num_object_clones":0,"num_object_copies":24804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6639","reported_seq":6704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.672728+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:58.672728+0000","last_peered":"2024-09-11T04:59:58.672728+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:58.672728+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50366784,"num_objects":6149,"num_object_clones":0,"num_object_copies":24596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6636,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6520","reported_seq":6607,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.984189+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:59.984189+0000","last_peered":"2024-09-11T04:59:59.984189+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:59.984189+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6520,"log_dups_size":0,"ondisk_log_size":6520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49214624,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6518,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6571","reported_seq":6632,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.827954+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:58.827954+0000","last_peered":"2024-09-11T04:59:58.827954+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:58.827954+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49819376,"num_objects":6082,"num_object_clones":0,"num_object_copies":24328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6406","reported_seq":11001,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T04:59:58.324824+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T04:59:58.324824+0000","last_peered":"2024-09-11T04:59:58.324824+0000","last_clean":"2024-09-11T04:59:58.324824+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T04:59:58.324824+0000","last_undegraded":"2024-09-11T04:59:58.324824+0000","last_fullsized":"2024-09-11T04:59:58.324824+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48971776,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6637","reported_seq":6716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.989985+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:59.989985+0000","last_peered":"2024-09-11T04:59:59.989985+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:59.989985+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6637,"log_dups_size":0,"ondisk_log_size":6637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49855056,"num_objects":6086,"num_object_clones":0,"num_object_copies":24344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6636,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6532","reported_seq":6587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.858163+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:58.858163+0000","last_peered":"2024-09-11T04:59:58.858163+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:58.858163+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49640608,"num_objects":6060,"num_object_clones":0,"num_object_copies":24240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6449","reported_seq":6542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.866382+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:59.866382+0000","last_peered":"2024-09-11T04:59:59.866382+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:59.866382+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821408,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6599","reported_seq":6667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:57.300158+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T04:59:57.300158+0000","last_peered":"2024-09-11T04:59:57.300158+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T04:59:57.300158+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50607264,"num_objects":6178,"num_object_clones":0,"num_object_copies":24712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6597,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":795338376,"num_objects":97093,"num_object_clones":0,"num_object_copies":388372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98100,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2105344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1462468608,"data_stored":1462468608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104316,"ondisk_log_size":104316,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745416,"kb_used_data":388856,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93626424,"statfs":{"total":96636764160,"available":95873458176,"internally_reserved":0,"allocated":398188544,"data_stored":398008575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,6,3,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756616,"kb_used_data":388088,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":93615224,"statfs":{"total":96636764160,"available":95861989376,"internally_reserved":0,"allocated":397402112,"data_stored":397222143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,0,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534236,"kb_used_data":263820,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93837604,"statfs":{"total":96636764160,"available":96089706496,"internally_reserved":0,"allocated":270151680,"data_stored":269971711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276885202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755348,"kb_used_data":390520,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93616492,"statfs":{"total":96636764160,"available":95863287808,"internally_reserved":0,"allocated":399892480,"data_stored":399712511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":398897152,"data_stored":398897152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269864960,"data_stored":269864960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396283904,"data_stored":396283904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397422592,"data_stored":397422592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:03.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:03.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6481", 'reported_seq': 6527, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:58.884401+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T04:59:58.884401+0000', 'last_peered': '2024-09-11T04:59:58.884401+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T04:59:58.884401+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6481, 'log_dups_size': 0, 'ondisk_log_size': 6481, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49422336, 'num_objects': 6033, 'num_object_clones': 0, 'num_object_copies': 24132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6481, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3721}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:03.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:03.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6557", 'reported_seq': 6624, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:57.364662+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T04:59:57.364662+0000', 'last_peered': '2024-09-11T04:59:57.364662+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T04:59:57.364662+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6557, 'log_dups_size': 0, 'ondisk_log_size': 6557, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50631840, 'num_objects': 6181, 'num_object_clones': 0, 'num_object_copies': 24724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6555, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:03.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6314", 'reported_seq': 6403, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:59.834518+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T04:59:59.834518+0000', 'last_peered': '2024-09-11T04:59:59.834518+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T04:59:59.834518+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6314, 'log_dups_size': 0, 'ondisk_log_size': 6314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47889000, 'num_objects': 5847, 'num_object_clones': 0, 'num_object_copies': 23388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2217, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5847, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6313, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3631}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6399", 'reported_seq': 6467, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:57.119767+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T04:59:57.119767+0000', 'last_peered': '2024-09-11T04:59:57.119767+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T04:59:57.119767+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6399, 'log_dups_size': 0, 'ondisk_log_size': 6399, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49288352, 'num_objects': 6017, 'num_object_clones': 0, 'num_object_copies': 24068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6397, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3739}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6471", 'reported_seq': 6532, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:58.790732+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T04:59:58.790732+0000', 'last_peered': '2024-09-11T04:59:58.790732+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T04:59:58.790732+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6471, 'log_dups_size': 0, 'ondisk_log_size': 6471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49640608, 'num_objects': 6060, 'num_object_clones': 0, 'num_object_copies': 24240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6469, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6555", 'reported_seq': 6602, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:58.836530+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T04:59:58.836530+0000', 'last_peered': '2024-09-11T04:59:58.836530+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T04:59:58.836530+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6555, 'log_dups_size': 0, 'ondisk_log_size': 6555, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49848320, 'num_objects': 6085, 'num_object_clones': 0, 'num_object_copies': 24340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6555, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3765}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6562", 'reported_seq': 6630, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:57.215818+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T04:59:57.215818+0000', 'last_peered': '2024-09-11T04:59:57.215818+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T04:59:57.215818+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6562, 'log_dups_size': 0, 'ondisk_log_size': 6562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50525344, 'num_objects': 6168, 'num_object_clones': 0, 'num_object_copies': 24672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6560, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3851}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:03.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6624", 'reported_seq': 7204, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:57.416323+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T04:59:57.416323+0000', 'last_peered': '2024-09-11T04:59:57.416323+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T04:59:57.416323+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6624, 'log_dups_size': 0, 'ondisk_log_size': 6624, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50795680, 'num_objects': 6201, 'num_object_clones': 0, 'num_object_copies': 24804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6622, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 259, 'num_bytes_recovered': 2105344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4097}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6639", 'reported_seq': 6704, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:58.672728+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T04:59:58.672728+0000', 'last_peered': '2024-09-11T04:59:58.672728+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T04:59:58.672728+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6639, 'log_dups_size': 0, 'ondisk_log_size': 6639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50366784, 'num_objects': 6149, 'num_object_clones': 0, 'num_object_copies': 24596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2395, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6636, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3757}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6520", 'reported_seq': 6607, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:59.984189+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T04:59:59.984189+0000', 'last_peered': '2024-09-11T04:59:59.984189+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T04:59:59.984189+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6520, 'log_dups_size': 0, 'ondisk_log_size': 6520, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49214624, 'num_objects': 6008, 'num_object_clones': 0, 'num_object_copies': 24032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6518, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3693}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6571", 'reported_seq': 6632, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:58.827954+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T04:59:58.827954+0000', 'last_peered': '2024-09-11T04:59:58.827954+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T04:59:58.827954+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6571, 'log_dups_size': 0, 'ondisk_log_size': 6571, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49819376, 'num_objects': 6082, 'num_object_clones': 0, 'num_object_copies': 24328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6568, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3759}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6637", 'reported_seq': 6716, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:59.989985+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T04:59:59.989985+0000', 'last_peered': '2024-09-11T04:59:59.989985+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T04:59:59.989985+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6637, 'log_dups_size': 0, 'ondisk_log_size': 6637, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49855056, 'num_objects': 6086, 'num_object_clones': 0, 'num_object_copies': 24344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6636, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:03.313 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:03.314 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6532", 'reported_seq': 6587, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:58.858163+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T04:59:58.858163+0000', 'last_peered': '2024-09-11T04:59:58.858163+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T04:59:58.858163+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6532, 'log_dups_size': 0, 'ondisk_log_size': 6532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49640608, 'num_objects': 6060, 'num_object_clones': 0, 'num_object_copies': 24240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6530, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:03.314 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:03.314 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6449", 'reported_seq': 6542, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:59.866382+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T04:59:59.866382+0000', 'last_peered': '2024-09-11T04:59:59.866382+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T04:59:59.866382+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6449, 'log_dups_size': 0, 'ondisk_log_size': 6449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48821408, 'num_objects': 5960, 'num_object_clones': 0, 'num_object_copies': 23840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6447, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3647}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:03.314 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:03.314 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6599", 'reported_seq': 6667, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T04:59:57.300158+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T04:59:57.300158+0000', 'last_peered': '2024-09-11T04:59:57.300158+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T04:59:57.300158+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6599, 'log_dups_size': 0, 'ondisk_log_size': 6599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50607264, 'num_objects': 6178, 'num_object_clones': 0, 'num_object_copies': 24712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6597, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3867}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:03.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:04.054 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34372/388372 objects degraded (8.850%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":331,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":97093,"data_bytes":795338376,"bytes_used":2858614784,"bytes_avail":383688441856,"bytes_total":386547056640,"degraded_objects":34372,"degraded_total":388372,"degraded_ratio":0.088502775689287591,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":58004,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":696320,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":245},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:04.091 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:04.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:04.718 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:04.719 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:04.748 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-11T05:00:03.706541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791422600,"num_objects":96615,"num_object_clones":0,"num_object_copies":386460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98578,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":2539520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104794,"ondisk_log_size":104794,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781912,"kb_used_data":1425740,"kb_used_omap":14,"kb_used_meta":1356081,"kb_avail":374705448,"statfs":{"total":386547056640,"available":383698378752,"internally_reserved":0,"allocated":1459957760,"data_stored":1459237884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1388627510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,2,13,15,15,3,0,0,2,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25305088,"num_objects":-3089,"num_object_clones":0,"num_object_copies":-12356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3089,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004843"},"pg_stats":[{"pgid":"1.f","version":"33'6481","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.884401+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T04:59:58.884401+0000","last_peered":"2024-09-11T04:59:58.884401+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T04:59:58.884401+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6481,"log_dups_size":0,"ondisk_log_size":6481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49422336,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6481,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6656","reported_seq":6729,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:02.240077+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:02.240077+0000","last_peered":"2024-09-11T05:00:02.240077+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:02.240077+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6656,"log_dups_size":0,"ondisk_log_size":6656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49820832,"num_objects":6082,"num_object_clones":0,"num_object_copies":24328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6314","reported_seq":6403,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.834518+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T04:59:59.834518+0000","last_peered":"2024-09-11T04:59:59.834518+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T04:59:59.834518+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47889000,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6313,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3631}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6480","reported_seq":6556,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:02.201971+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:02.201971+0000","last_peered":"2024-09-11T05:00:02.201971+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:02.201971+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48624800,"num_objects":5936,"num_object_clones":0,"num_object_copies":23744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6478,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6471","reported_seq":6532,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.790732+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T04:59:58.790732+0000","last_peered":"2024-09-11T04:59:58.790732+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T04:59:58.790732+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49640608,"num_objects":6060,"num_object_clones":0,"num_object_copies":24240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6469,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6555","reported_seq":6602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.836530+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T04:59:58.836530+0000","last_peered":"2024-09-11T04:59:58.836530+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T04:59:58.836530+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6555,"log_dups_size":0,"ondisk_log_size":6555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49848320,"num_objects":6085,"num_object_clones":0,"num_object_copies":24340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6633","reported_seq":6703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:02.371716+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:02.371716+0000","last_peered":"2024-09-11T05:00:02.371716+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:02.371716+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49943712,"num_objects":6097,"num_object_clones":0,"num_object_copies":24388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6631,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3781}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6716","reported_seq":7404,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:02.282971+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:02.282971+0000","last_peered":"2024-09-11T05:00:02.282971+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:02.282971+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50042016,"num_objects":6109,"num_object_clones":0,"num_object_copies":24436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":2539520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6639","reported_seq":6704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.672728+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T04:59:58.672728+0000","last_peered":"2024-09-11T04:59:58.672728+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T04:59:58.672728+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50366784,"num_objects":6149,"num_object_clones":0,"num_object_copies":24596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6636,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6520","reported_seq":6607,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.984189+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T04:59:59.984189+0000","last_peered":"2024-09-11T04:59:59.984189+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T04:59:59.984189+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6520,"log_dups_size":0,"ondisk_log_size":6520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49214624,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6518,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6571","reported_seq":6632,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.827954+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T04:59:58.827954+0000","last_peered":"2024-09-11T04:59:58.827954+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T04:59:58.827954+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49819376,"num_objects":6082,"num_object_clones":0,"num_object_copies":24328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6477","reported_seq":11072,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:03.233403+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:03.233403+0000","last_peered":"2024-09-11T05:00:03.233403+0000","last_clean":"2024-09-11T05:00:03.233403+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:03.233403+0000","last_undegraded":"2024-09-11T05:00:03.233403+0000","last_fullsized":"2024-09-11T05:00:03.233403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48390144,"num_objects":5907,"num_object_clones":0,"num_object_copies":23628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6637","reported_seq":6716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.989985+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T04:59:59.989985+0000","last_peered":"2024-09-11T04:59:59.989985+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T04:59:59.989985+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6637,"log_dups_size":0,"ondisk_log_size":6637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49855056,"num_objects":6086,"num_object_clones":0,"num_object_copies":24344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6636,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6532","reported_seq":6587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:58.858163+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T04:59:58.858163+0000","last_peered":"2024-09-11T04:59:58.858163+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T04:59:58.858163+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49640608,"num_objects":6060,"num_object_clones":0,"num_object_copies":24240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6449","reported_seq":6542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T04:59:59.866382+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T04:59:59.866382+0000","last_peered":"2024-09-11T04:59:59.866382+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T04:59:59.866382+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821408,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3647}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6663","reported_seq":6731,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:02.325303+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:02.325303+0000","last_peered":"2024-09-11T05:00:02.325303+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:02.325303+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50082976,"num_objects":6114,"num_object_clones":0,"num_object_copies":24456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6661,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791422600,"num_objects":96615,"num_object_clones":0,"num_object_copies":386460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98578,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":2539520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1456848896,"data_stored":1456848896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104794,"ondisk_log_size":104794,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745416,"kb_used_data":388856,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93626424,"statfs":{"total":96636764160,"available":95873458176,"internally_reserved":0,"allocated":398188544,"data_stored":398008575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,6,3,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756616,"kb_used_data":388088,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":93615224,"statfs":{"total":96636764160,"available":95861989376,"internally_reserved":0,"allocated":397402112,"data_stored":397222143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,0,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542120,"kb_used_data":263384,"kb_used_omap":4,"kb_used_meta":278715,"kb_avail":93829720,"statfs":{"total":96636764160,"available":96081633280,"internally_reserved":0,"allocated":269705216,"data_stored":269525247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":285404882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,6,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737760,"kb_used_data":385412,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93634080,"statfs":{"total":96636764160,"available":95881297920,"internally_reserved":0,"allocated":394661888,"data_stored":394481919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,3,3,7,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393740288,"data_stored":393740288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269402112,"data_stored":269402112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396283904,"data_stored":396283904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397422592,"data_stored":397422592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:07.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:08.393 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:08.393 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:08.432 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-11T05:00:07.708434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":781829768,"num_objects":95444,"num_object_clones":0,"num_object_copies":381776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99749,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105965,"ondisk_log_size":105965,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762836,"kb_used_data":1410820,"kb_used_omap":14,"kb_used_meta":1351921,"kb_avail":374724524,"statfs":{"total":386547056640,"available":383717912576,"internally_reserved":0,"allocated":1444679680,"data_stored":1443959804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1384367670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,2,16,9,13,5,1,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24100864,"num_objects":-2942,"num_object_clones":0,"num_object_copies":-11768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2942,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005185"},"pg_stats":[{"pgid":"1.f","version":"33'6554","reported_seq":6600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.909560+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:03.909560+0000","last_peered":"2024-09-11T05:00:03.909560+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:03.909560+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6554,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6736","reported_seq":6809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.440211+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:07.440211+0000","last_peered":"2024-09-11T05:00:07.440211+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:07.440211+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49165472,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6403","reported_seq":6492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.933382+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:04.933382+0000","last_peered":"2024-09-11T05:00:04.933382+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:04.933382+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47159912,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6402,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3542}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6564","reported_seq":6640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.466633+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:07.466633+0000","last_peered":"2024-09-11T05:00:07.466633+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:07.466633+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6564,"log_dups_size":0,"ondisk_log_size":6564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936672,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6562,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6542","reported_seq":6602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.900173+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:03.900173+0000","last_peered":"2024-09-11T05:00:03.900173+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:03.900173+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49058976,"num_objects":5989,"num_object_clones":0,"num_object_copies":23956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6540,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6633","reported_seq":6681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.689098+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:03.689098+0000","last_peered":"2024-09-11T05:00:03.689098+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:03.689098+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49209344,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6689","reported_seq":6759,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.354846+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:07.354846+0000","last_peered":"2024-09-11T05:00:07.354846+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:07.354846+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49484960,"num_objects":6041,"num_object_clones":0,"num_object_copies":24164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6811","reported_seq":7575,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.422029+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:07.422029+0000","last_peered":"2024-09-11T05:00:07.422029+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:07.422029+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6811,"log_dups_size":0,"ondisk_log_size":6811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6716","reported_seq":6781,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.931025+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:03.931025+0000","last_peered":"2024-09-11T05:00:03.931025+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:03.931025+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49736000,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6713,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6593","reported_seq":6681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.966927+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:04.966927+0000","last_peered":"2024-09-11T05:00:04.966927+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:04.966927+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48616608,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6591,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6658","reported_seq":6719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.912869+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:03.912869+0000","last_peered":"2024-09-11T05:00:03.912869+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:03.912869+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6658,"log_dups_size":0,"ondisk_log_size":6658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49106672,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6655,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6477","reported_seq":11072,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:03.233403+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:03.233403+0000","last_peered":"2024-09-11T05:00:03.233403+0000","last_clean":"2024-09-11T05:00:03.233403+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:03.233403+0000","last_undegraded":"2024-09-11T05:00:03.233403+0000","last_fullsized":"2024-09-11T05:00:03.233403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48390144,"num_objects":5907,"num_object_clones":0,"num_object_copies":23628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6707","reported_seq":6786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.740182+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:04.740182+0000","last_peered":"2024-09-11T05:00:04.740182+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:04.740182+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49281616,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6598","reported_seq":6653,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.741996+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:03.741996+0000","last_peered":"2024-09-11T05:00:03.741996+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:03.741996+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6598,"log_dups_size":0,"ondisk_log_size":6598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49099936,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6521","reported_seq":6614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.890164+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:04.890164+0000","last_peered":"2024-09-11T05:00:04.890164+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:04.890164+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6521,"log_dups_size":0,"ondisk_log_size":6521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48231584,"num_objects":5888,"num_object_clones":0,"num_object_copies":23552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6763","reported_seq":6830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.452949+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:07.452949+0000","last_peered":"2024-09-11T05:00:07.452949+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:07.452949+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6763,"log_dups_size":0,"ondisk_log_size":6763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6761,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3703}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":781829768,"num_objects":95444,"num_object_clones":0,"num_object_copies":381776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99749,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1441402880,"data_stored":1441402880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105965,"ondisk_log_size":105965,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744524,"kb_used_data":383804,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93627316,"statfs":{"total":96636764160,"available":95874371584,"internally_reserved":0,"allocated":393015296,"data_stored":392835327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,2,6,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739192,"kb_used_data":383144,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93632648,"statfs":{"total":96636764160,"available":95879831552,"internally_reserved":0,"allocated":392339456,"data_stored":392159487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,1,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542120,"kb_used_data":263384,"kb_used_omap":4,"kb_used_meta":278715,"kb_avail":93829720,"statfs":{"total":96636764160,"available":96081633280,"internally_reserved":0,"allocated":269705216,"data_stored":269525247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":285404882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,6,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737000,"kb_used_data":380488,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93634840,"statfs":{"total":96636764160,"available":95882076160,"internally_reserved":0,"allocated":389619712,"data_stored":389439743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,8,2,1,3,1,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388505600,"data_stored":388505600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269402112,"data_stored":269402112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391233536,"data_stored":391233536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":392261632,"data_stored":392261632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:08.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:09.025 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:09.026 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:09.045 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-11T05:00:07.708434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":781829768,"num_objects":95444,"num_object_clones":0,"num_object_copies":381776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99749,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105965,"ondisk_log_size":105965,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762836,"kb_used_data":1410820,"kb_used_omap":14,"kb_used_meta":1351921,"kb_avail":374724524,"statfs":{"total":386547056640,"available":383717912576,"internally_reserved":0,"allocated":1444679680,"data_stored":1443959804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1384367670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,2,16,9,13,5,1,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24100864,"num_objects":-2942,"num_object_clones":0,"num_object_copies":-11768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2942,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005185"},"pg_stats":[{"pgid":"1.f","version":"33'6554","reported_seq":6600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.909560+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:03.909560+0000","last_peered":"2024-09-11T05:00:03.909560+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:03.909560+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6554,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6736","reported_seq":6809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.440211+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:07.440211+0000","last_peered":"2024-09-11T05:00:07.440211+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:07.440211+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49165472,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6403","reported_seq":6492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.933382+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:04.933382+0000","last_peered":"2024-09-11T05:00:04.933382+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:04.933382+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47159912,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6402,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3542}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6564","reported_seq":6640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.466633+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:07.466633+0000","last_peered":"2024-09-11T05:00:07.466633+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:07.466633+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6564,"log_dups_size":0,"ondisk_log_size":6564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936672,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6562,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6542","reported_seq":6602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.900173+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:03.900173+0000","last_peered":"2024-09-11T05:00:03.900173+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:03.900173+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49058976,"num_objects":5989,"num_object_clones":0,"num_object_copies":23956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6540,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6633","reported_seq":6681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.689098+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:03.689098+0000","last_peered":"2024-09-11T05:00:03.689098+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:03.689098+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49209344,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6689","reported_seq":6759,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.354846+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:07.354846+0000","last_peered":"2024-09-11T05:00:07.354846+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:07.354846+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49484960,"num_objects":6041,"num_object_clones":0,"num_object_copies":24164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6811","reported_seq":7575,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.422029+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:07.422029+0000","last_peered":"2024-09-11T05:00:07.422029+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:07.422029+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6811,"log_dups_size":0,"ondisk_log_size":6811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6716","reported_seq":6781,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.931025+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:03.931025+0000","last_peered":"2024-09-11T05:00:03.931025+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:03.931025+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49736000,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6713,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6593","reported_seq":6681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.966927+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:04.966927+0000","last_peered":"2024-09-11T05:00:04.966927+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:04.966927+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48616608,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6591,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6658","reported_seq":6719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.912869+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:03.912869+0000","last_peered":"2024-09-11T05:00:03.912869+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:03.912869+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6658,"log_dups_size":0,"ondisk_log_size":6658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49106672,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6655,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6477","reported_seq":11072,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:03.233403+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:03.233403+0000","last_peered":"2024-09-11T05:00:03.233403+0000","last_clean":"2024-09-11T05:00:03.233403+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:03.233403+0000","last_undegraded":"2024-09-11T05:00:03.233403+0000","last_fullsized":"2024-09-11T05:00:03.233403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48390144,"num_objects":5907,"num_object_clones":0,"num_object_copies":23628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6707","reported_seq":6786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.740182+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:04.740182+0000","last_peered":"2024-09-11T05:00:04.740182+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:04.740182+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49281616,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6598","reported_seq":6653,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:03.741996+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:03.741996+0000","last_peered":"2024-09-11T05:00:03.741996+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:03.741996+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6598,"log_dups_size":0,"ondisk_log_size":6598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49099936,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6521","reported_seq":6614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.890164+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:04.890164+0000","last_peered":"2024-09-11T05:00:04.890164+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:04.890164+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6521,"log_dups_size":0,"ondisk_log_size":6521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48231584,"num_objects":5888,"num_object_clones":0,"num_object_copies":23552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6763","reported_seq":6830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.452949+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:07.452949+0000","last_peered":"2024-09-11T05:00:07.452949+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:07.452949+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6763,"log_dups_size":0,"ondisk_log_size":6763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6761,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3703}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":781829768,"num_objects":95444,"num_object_clones":0,"num_object_copies":381776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99749,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1441402880,"data_stored":1441402880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105965,"ondisk_log_size":105965,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744524,"kb_used_data":383804,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93627316,"statfs":{"total":96636764160,"available":95874371584,"internally_reserved":0,"allocated":393015296,"data_stored":392835327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,2,6,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739192,"kb_used_data":383144,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93632648,"statfs":{"total":96636764160,"available":95879831552,"internally_reserved":0,"allocated":392339456,"data_stored":392159487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,1,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542120,"kb_used_data":263384,"kb_used_omap":4,"kb_used_meta":278715,"kb_avail":93829720,"statfs":{"total":96636764160,"available":96081633280,"internally_reserved":0,"allocated":269705216,"data_stored":269525247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":285404882},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,6,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737000,"kb_used_data":380488,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93634840,"statfs":{"total":96636764160,"available":95882076160,"internally_reserved":0,"allocated":389619712,"data_stored":389439743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,8,2,1,3,1,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388505600,"data_stored":388505600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269402112,"data_stored":269402112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391233536,"data_stored":391233536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":392261632,"data_stored":392261632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:09.046 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:09.046 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6554", 'reported_seq': 6600, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:03.909560+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:03.909560+0000', 'last_peered': '2024-09-11T05:00:03.909560+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:03.909560+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6554, 'log_dups_size': 0, 'ondisk_log_size': 6554, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48824320, 'num_objects': 5960, 'num_object_clones': 0, 'num_object_copies': 23840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6554, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3648}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:09.046 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:09.046 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6736", 'reported_seq': 6809, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:07.440211+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:07.440211+0000', 'last_peered': '2024-09-11T05:00:07.440211+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:07.440211+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6736, 'log_dups_size': 0, 'ondisk_log_size': 6736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49165472, 'num_objects': 6002, 'num_object_clones': 0, 'num_object_copies': 24008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2278, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:09.046 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:09.046 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6403", 'reported_seq': 6492, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:04.933382+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:04.933382+0000', 'last_peered': '2024-09-11T05:00:04.933382+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:04.933382+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6403, 'log_dups_size': 0, 'ondisk_log_size': 6403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47159912, 'num_objects': 5758, 'num_object_clones': 0, 'num_object_copies': 23032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2217, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5758, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6402, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3542}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6564", 'reported_seq': 6640, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:07.466633+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:07.466633+0000', 'last_peered': '2024-09-11T05:00:07.466633+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:07.466633+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6564, 'log_dups_size': 0, 'ondisk_log_size': 6564, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47936672, 'num_objects': 5852, 'num_object_clones': 0, 'num_object_copies': 23408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6562, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6542", 'reported_seq': 6602, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:03.900173+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:03.900173+0000', 'last_peered': '2024-09-11T05:00:03.900173+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:03.900173+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6542, 'log_dups_size': 0, 'ondisk_log_size': 6542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49058976, 'num_objects': 5989, 'num_object_clones': 0, 'num_object_copies': 23956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2251, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6540, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3740}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6633", 'reported_seq': 6681, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:03.689098+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:03.689098+0000', 'last_peered': '2024-09-11T05:00:03.689098+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:03.689098+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6633, 'log_dups_size': 0, 'ondisk_log_size': 6633, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49209344, 'num_objects': 6007, 'num_object_clones': 0, 'num_object_copies': 24028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2320, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6633, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3687}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6689", 'reported_seq': 6759, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:07.354846+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:07.354846+0000', 'last_peered': '2024-09-11T05:00:07.354846+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:07.354846+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6689, 'log_dups_size': 0, 'ondisk_log_size': 6689, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49484960, 'num_objects': 6041, 'num_object_clones': 0, 'num_object_copies': 24164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2318, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6687, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3725}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:09.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6811", 'reported_seq': 7575, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:07.422029+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:07.422029+0000', 'last_peered': '2024-09-11T05:00:07.422029+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:07.422029+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6811, 'log_dups_size': 0, 'ondisk_log_size': 6811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49263776, 'num_objects': 6014, 'num_object_clones': 0, 'num_object_copies': 24056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6809, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 350, 'num_bytes_recovered': 2850816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4001}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6716", 'reported_seq': 6781, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:03.931025+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:03.931025+0000', 'last_peered': '2024-09-11T05:00:03.931025+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:03.931025+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6716, 'log_dups_size': 0, 'ondisk_log_size': 6716, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49736000, 'num_objects': 6072, 'num_object_clones': 0, 'num_object_copies': 24288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2395, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6713, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3680}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6593", 'reported_seq': 6681, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:04.966927+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:04.966927+0000', 'last_peered': '2024-09-11T05:00:04.966927+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:04.966927+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6593, 'log_dups_size': 0, 'ondisk_log_size': 6593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48616608, 'num_objects': 5935, 'num_object_clones': 0, 'num_object_copies': 23740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6591, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3620}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6658", 'reported_seq': 6719, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:03.912869+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:03.912869+0000', 'last_peered': '2024-09-11T05:00:03.912869+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:03.912869+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6658, 'log_dups_size': 0, 'ondisk_log_size': 6658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49106672, 'num_objects': 5995, 'num_object_clones': 0, 'num_object_copies': 23980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6655, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3672}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6707", 'reported_seq': 6786, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:04.740182+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:04.740182+0000', 'last_peered': '2024-09-11T05:00:04.740182+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:04.740182+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6707, 'log_dups_size': 0, 'ondisk_log_size': 6707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49281616, 'num_objects': 6016, 'num_object_clones': 0, 'num_object_copies': 24064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6706, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3684}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:09.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:09.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6598", 'reported_seq': 6653, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:03.741996+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:03.741996+0000', 'last_peered': '2024-09-11T05:00:03.741996+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:03.741996+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6598, 'log_dups_size': 0, 'ondisk_log_size': 6598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49099936, 'num_objects': 5994, 'num_object_clones': 0, 'num_object_copies': 23976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6596, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3707}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:09.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:09.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6521", 'reported_seq': 6614, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:04.890164+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:04.890164+0000', 'last_peered': '2024-09-11T05:00:04.890164+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:04.890164+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6521, 'log_dups_size': 0, 'ondisk_log_size': 6521, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48231584, 'num_objects': 5888, 'num_object_clones': 0, 'num_object_copies': 23552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6519, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:09.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:09.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6763", 'reported_seq': 6830, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:07.452949+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:07.452949+0000', 'last_peered': '2024-09-11T05:00:07.452949+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:07.452949+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6763, 'log_dups_size': 0, 'ondisk_log_size': 6763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49263776, 'num_objects': 6014, 'num_object_clones': 0, 'num_object_copies': 24056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6761, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:09.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:09.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:09.813 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34273/381776 objects degraded (8.977%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":336,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95444,"data_bytes":781829768,"bytes_used":2829144064,"bytes_avail":383717912576,"bytes_total":386547056640,"degraded_objects":34273,"degraded_total":381776,"degraded_ratio":0.089772536775491393,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":88708,"recovering_keys_per_sec":0,"num_objects_recovered":141,"num_bytes_recovered":1064960,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":245},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:09.844 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:09.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:10.481 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:10.504 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-11T05:00:09.709303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":777242248,"num_objects":94884,"num_object_clones":0,"num_object_copies":379536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100309,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106525,"ondisk_log_size":106525,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767300,"kb_used_data":1405552,"kb_used_omap":14,"kb_used_meta":1361649,"kb_avail":374720060,"statfs":{"total":386547056640,"available":383713341440,"internally_reserved":0,"allocated":1439285248,"data_stored":1438565372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1394329142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,2,2,5,16,9,5,5,1,0,0,0,0,2,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25493504,"num_objects":-3112,"num_object_clones":0,"num_object_copies":-12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3112,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005359"},"pg_stats":[{"pgid":"1.f","version":"33'6628","reported_seq":6677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.948551+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:08.948551+0000","last_peered":"2024-09-11T05:00:08.948551+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:08.948551+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6628,"log_dups_size":0,"ondisk_log_size":6628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48218112,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6628,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6736","reported_seq":6809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.440211+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:07.440211+0000","last_peered":"2024-09-11T05:00:07.440211+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:07.440211+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49165472,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6403","reported_seq":6492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.933382+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:04.933382+0000","last_peered":"2024-09-11T05:00:04.933382+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:04.933382+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47159912,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6402,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3542}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6564","reported_seq":6640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.466633+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:07.466633+0000","last_peered":"2024-09-11T05:00:07.466633+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:07.466633+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6564,"log_dups_size":0,"ondisk_log_size":6564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47936672,"num_objects":5852,"num_object_clones":0,"num_object_copies":23408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6562,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6624","reported_seq":6693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.987191+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:08.987191+0000","last_peered":"2024-09-11T05:00:08.987191+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:08.987191+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48387232,"num_objects":5907,"num_object_clones":0,"num_object_copies":23628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6703","reported_seq":6755,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.877107+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:08.877107+0000","last_peered":"2024-09-11T05:00:08.877107+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:08.877107+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6703,"log_dups_size":0,"ondisk_log_size":6703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48635904,"num_objects":5937,"num_object_clones":0,"num_object_copies":23748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6689","reported_seq":6759,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.354846+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:07.354846+0000","last_peered":"2024-09-11T05:00:07.354846+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:07.354846+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49484960,"num_objects":6041,"num_object_clones":0,"num_object_copies":24164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3725}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6811","reported_seq":7575,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.422029+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:07.422029+0000","last_peered":"2024-09-11T05:00:07.422029+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:07.422029+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6811,"log_dups_size":0,"ondisk_log_size":6811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6792","reported_seq":6865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.920984+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:08.920984+0000","last_peered":"2024-09-11T05:00:08.920984+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:08.920984+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49113408,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6593","reported_seq":6681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.966927+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:04.966927+0000","last_peered":"2024-09-11T05:00:04.966927+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:04.966927+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48616608,"num_objects":5935,"num_object_clones":0,"num_object_copies":23740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6591,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6741","reported_seq":6808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.929814+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:08.929814+0000","last_peered":"2024-09-11T05:00:08.929814+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:08.929814+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6741,"log_dups_size":0,"ondisk_log_size":6741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48426736,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6563","reported_seq":11158,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:08.479173+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:08.479173+0000","last_peered":"2024-09-11T05:00:08.479173+0000","last_clean":"2024-09-11T05:00:08.479173+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:08.479173+0000","last_undegraded":"2024-09-11T05:00:08.479173+0000","last_fullsized":"2024-09-11T05:00:08.479173+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47685632,"num_objects":5821,"num_object_clones":0,"num_object_copies":23284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":373,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6707","reported_seq":6786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.740182+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:04.740182+0000","last_peered":"2024-09-11T05:00:04.740182+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:04.740182+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49281616,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6687","reported_seq":6742,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.942343+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:08.942343+0000","last_peered":"2024-09-11T05:00:08.942343+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:08.942343+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48370848,"num_objects":5905,"num_object_clones":0,"num_object_copies":23620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3618}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6521","reported_seq":6614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:04.890164+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:04.890164+0000","last_peered":"2024-09-11T05:00:04.890164+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:04.890164+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6521,"log_dups_size":0,"ondisk_log_size":6521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48231584,"num_objects":5888,"num_object_clones":0,"num_object_copies":23552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6763","reported_seq":6830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:07.452949+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:07.452949+0000","last_peered":"2024-09-11T05:00:07.452949+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:07.452949+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6763,"log_dups_size":0,"ondisk_log_size":6763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263776,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6761,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3703}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":777242248,"num_objects":94884,"num_object_clones":0,"num_object_copies":379536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100309,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1435922432,"data_stored":1435922432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106525,"ondisk_log_size":106525,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761756,"kb_used_data":378824,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93610084,"statfs":{"total":96636764160,"available":95856726016,"internally_reserved":0,"allocated":387915776,"data_stored":387735807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":392097488},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,2,0,8,4,3,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739192,"kb_used_data":383144,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93632648,"statfs":{"total":96636764160,"available":95879831552,"internally_reserved":0,"allocated":392339456,"data_stored":392159487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,1,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529352,"kb_used_data":263096,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93842488,"statfs":{"total":96636764160,"available":96094707712,"internally_reserved":0,"allocated":269410304,"data_stored":269230335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272625362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737000,"kb_used_data":380488,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93634840,"statfs":{"total":96636764160,"available":95882076160,"internally_reserved":0,"allocated":389619712,"data_stored":389439743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,8,2,1,3,1,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388505600,"data_stored":388505600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269123584,"data_stored":269123584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391233536,"data_stored":391233536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":387059712,"data_stored":387059712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:13.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:14.130 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:14.131 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:14.154 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-11T05:00:13.710932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":771098248,"num_objects":94134,"num_object_clones":0,"num_object_copies":376536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101059,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107275,"ondisk_log_size":107275,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805644,"kb_used_data":1395332,"kb_used_omap":14,"kb_used_meta":1410225,"kb_avail":374681716,"statfs":{"total":386547056640,"available":383674077184,"internally_reserved":0,"allocated":1428819968,"data_stored":1428100092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1444070966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,3,2,4,15,8,4,1,0,0,0,1,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24240128,"num_objects":-2959,"num_object_clones":0,"num_object_copies":-11836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2959,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005333"},"pg_stats":[{"pgid":"1.f","version":"33'6628","reported_seq":6677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.948551+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:08.948551+0000","last_peered":"2024-09-11T05:00:08.948551+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:08.948551+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6628,"log_dups_size":0,"ondisk_log_size":6628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48218112,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6628,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6813","reported_seq":6888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.308639+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:12.308639+0000","last_peered":"2024-09-11T05:00:12.308639+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:12.308639+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6813,"log_dups_size":0,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48534688,"num_objects":5925,"num_object_clones":0,"num_object_copies":23700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6473","reported_seq":6564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:09.810017+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:09.810017+0000","last_peered":"2024-09-11T05:00:09.810017+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:09.810017+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46586472,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6472,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3473}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6640","reported_seq":6716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.426902+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:12.426902+0000","last_peered":"2024-09-11T05:00:12.426902+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:12.426902+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6640,"log_dups_size":0,"ondisk_log_size":6640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314080,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6624","reported_seq":6693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.987191+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:08.987191+0000","last_peered":"2024-09-11T05:00:08.987191+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:08.987191+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48387232,"num_objects":5907,"num_object_clones":0,"num_object_copies":23628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6703","reported_seq":6755,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.877107+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:08.877107+0000","last_peered":"2024-09-11T05:00:08.877107+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:08.877107+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6703,"log_dups_size":0,"ondisk_log_size":6703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48635904,"num_objects":5937,"num_object_clones":0,"num_object_copies":23748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6767","reported_seq":6837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.385534+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:12.385534+0000","last_peered":"2024-09-11T05:00:12.385534+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:12.385534+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6767,"log_dups_size":0,"ondisk_log_size":6767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48845984,"num_objects":5963,"num_object_clones":0,"num_object_copies":23852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3647}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6899","reported_seq":7764,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.502866+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:12.502866+0000","last_peered":"2024-09-11T05:00:12.502866+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:12.502866+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":401,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3964}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6792","reported_seq":6865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.920984+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:08.920984+0000","last_peered":"2024-09-11T05:00:08.920984+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:08.920984+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49113408,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6670","reported_seq":6758,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:09.859032+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:09.859032+0000","last_peered":"2024-09-11T05:00:09.859032+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:09.859032+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6670,"log_dups_size":0,"ondisk_log_size":6670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47985824,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6668,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3543}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6741","reported_seq":6808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.929814+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:08.929814+0000","last_peered":"2024-09-11T05:00:08.929814+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:08.929814+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6741,"log_dups_size":0,"ondisk_log_size":6741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48426736,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6632","reported_seq":11227,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:13.483776+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:13.483776+0000","last_peered":"2024-09-11T05:00:13.483776+0000","last_clean":"2024-09-11T05:00:13.483776+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:13.483776+0000","last_undegraded":"2024-09-11T05:00:13.483776+0000","last_fullsized":"2024-09-11T05:00:13.483776+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47120384,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":442,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6785","reported_seq":6864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:09.817510+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:09.817510+0000","last_peered":"2024-09-11T05:00:09.817510+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:09.817510+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48642640,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3606}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6687","reported_seq":6742,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.942343+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:08.942343+0000","last_peered":"2024-09-11T05:00:08.942343+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:08.942343+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48370848,"num_objects":5905,"num_object_clones":0,"num_object_copies":23620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3618}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6591","reported_seq":6688,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:09.834178+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:09.834178+0000","last_peered":"2024-09-11T05:00:09.834178+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:09.834178+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6591,"log_dups_size":0,"ondisk_log_size":6591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47658144,"num_objects":5818,"num_object_clones":0,"num_object_copies":23272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6589,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6830","reported_seq":6900,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.438418+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:12.438418+0000","last_peered":"2024-09-11T05:00:12.438418+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:12.438418+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48714912,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":771098248,"num_objects":94134,"num_object_clones":0,"num_object_copies":376536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101059,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1425649664,"data_stored":1425649664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107275,"ondisk_log_size":107275,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761756,"kb_used_data":378824,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93610084,"statfs":{"total":96636764160,"available":95856726016,"internally_reserved":0,"allocated":387915776,"data_stored":387735807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":392097488},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,2,0,8,4,3,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756308,"kb_used_data":378052,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93615532,"statfs":{"total":96636764160,"available":95862304768,"internally_reserved":0,"allocated":387125248,"data_stored":386945279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,1,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533288,"kb_used_data":262872,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93838552,"statfs":{"total":96636764160,"available":96090677248,"internally_reserved":0,"allocated":269180928,"data_stored":269000959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276885202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754292,"kb_used_data":375584,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93617548,"statfs":{"total":96636764160,"available":95864369152,"internally_reserved":0,"allocated":384598016,"data_stored":384418047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":387772111},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,1,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383578112,"data_stored":383578112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268886016,"data_stored":268886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386125824,"data_stored":386125824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":387059712,"data_stored":387059712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:14.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:14.796 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:14.798 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:14.823 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-11T05:00:13.710932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":771098248,"num_objects":94134,"num_object_clones":0,"num_object_copies":376536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101059,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107275,"ondisk_log_size":107275,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805644,"kb_used_data":1395332,"kb_used_omap":14,"kb_used_meta":1410225,"kb_avail":374681716,"statfs":{"total":386547056640,"available":383674077184,"internally_reserved":0,"allocated":1428819968,"data_stored":1428100092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1444070966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,3,2,4,15,8,4,1,0,0,0,1,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24240128,"num_objects":-2959,"num_object_clones":0,"num_object_copies":-11836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2959,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005333"},"pg_stats":[{"pgid":"1.f","version":"33'6628","reported_seq":6677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.948551+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:08.948551+0000","last_peered":"2024-09-11T05:00:08.948551+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:08.948551+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6628,"log_dups_size":0,"ondisk_log_size":6628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48218112,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6628,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6813","reported_seq":6888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.308639+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:12.308639+0000","last_peered":"2024-09-11T05:00:12.308639+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:12.308639+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6813,"log_dups_size":0,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48534688,"num_objects":5925,"num_object_clones":0,"num_object_copies":23700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6473","reported_seq":6564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:09.810017+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:09.810017+0000","last_peered":"2024-09-11T05:00:09.810017+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:09.810017+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46586472,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6472,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3473}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6640","reported_seq":6716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.426902+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:12.426902+0000","last_peered":"2024-09-11T05:00:12.426902+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:12.426902+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6640,"log_dups_size":0,"ondisk_log_size":6640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314080,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6624","reported_seq":6693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.987191+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:08.987191+0000","last_peered":"2024-09-11T05:00:08.987191+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:08.987191+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48387232,"num_objects":5907,"num_object_clones":0,"num_object_copies":23628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6703","reported_seq":6755,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.877107+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:08.877107+0000","last_peered":"2024-09-11T05:00:08.877107+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:08.877107+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6703,"log_dups_size":0,"ondisk_log_size":6703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48635904,"num_objects":5937,"num_object_clones":0,"num_object_copies":23748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6767","reported_seq":6837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.385534+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:12.385534+0000","last_peered":"2024-09-11T05:00:12.385534+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:12.385534+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6767,"log_dups_size":0,"ondisk_log_size":6767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48845984,"num_objects":5963,"num_object_clones":0,"num_object_copies":23852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3647}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6899","reported_seq":7764,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.502866+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:12.502866+0000","last_peered":"2024-09-11T05:00:12.502866+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:12.502866+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":401,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3964}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6792","reported_seq":6865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.920984+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:08.920984+0000","last_peered":"2024-09-11T05:00:08.920984+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:08.920984+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49113408,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6670","reported_seq":6758,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:09.859032+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:09.859032+0000","last_peered":"2024-09-11T05:00:09.859032+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:09.859032+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6670,"log_dups_size":0,"ondisk_log_size":6670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47985824,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6668,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3543}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6741","reported_seq":6808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.929814+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:08.929814+0000","last_peered":"2024-09-11T05:00:08.929814+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:08.929814+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6741,"log_dups_size":0,"ondisk_log_size":6741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48426736,"num_objects":5912,"num_object_clones":0,"num_object_copies":23648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6632","reported_seq":11227,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:13.483776+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:13.483776+0000","last_peered":"2024-09-11T05:00:13.483776+0000","last_clean":"2024-09-11T05:00:13.483776+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:13.483776+0000","last_undegraded":"2024-09-11T05:00:13.483776+0000","last_fullsized":"2024-09-11T05:00:13.483776+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47120384,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":442,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6785","reported_seq":6864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:09.817510+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:09.817510+0000","last_peered":"2024-09-11T05:00:09.817510+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:09.817510+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48642640,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3606}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6687","reported_seq":6742,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:08.942343+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:08.942343+0000","last_peered":"2024-09-11T05:00:08.942343+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:08.942343+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48370848,"num_objects":5905,"num_object_clones":0,"num_object_copies":23620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3618}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6591","reported_seq":6688,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:09.834178+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:09.834178+0000","last_peered":"2024-09-11T05:00:09.834178+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:09.834178+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6591,"log_dups_size":0,"ondisk_log_size":6591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47658144,"num_objects":5818,"num_object_clones":0,"num_object_copies":23272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6589,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6830","reported_seq":6900,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.438418+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:12.438418+0000","last_peered":"2024-09-11T05:00:12.438418+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:12.438418+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48714912,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":771098248,"num_objects":94134,"num_object_clones":0,"num_object_copies":376536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101059,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1425649664,"data_stored":1425649664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107275,"ondisk_log_size":107275,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761756,"kb_used_data":378824,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93610084,"statfs":{"total":96636764160,"available":95856726016,"internally_reserved":0,"allocated":387915776,"data_stored":387735807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":392097488},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,2,0,8,4,3,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756308,"kb_used_data":378052,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93615532,"statfs":{"total":96636764160,"available":95862304768,"internally_reserved":0,"allocated":387125248,"data_stored":386945279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,1,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533288,"kb_used_data":262872,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93838552,"statfs":{"total":96636764160,"available":96090677248,"internally_reserved":0,"allocated":269180928,"data_stored":269000959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276885202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754292,"kb_used_data":375584,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93617548,"statfs":{"total":96636764160,"available":95864369152,"internally_reserved":0,"allocated":384598016,"data_stored":384418047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":387772111},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,1,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383578112,"data_stored":383578112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268886016,"data_stored":268886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386125824,"data_stored":386125824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":387059712,"data_stored":387059712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:14.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:14.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6628", 'reported_seq': 6677, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:08.948551+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:08.948551+0000', 'last_peered': '2024-09-11T05:00:08.948551+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:08.948551+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6628, 'log_dups_size': 0, 'ondisk_log_size': 6628, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48218112, 'num_objects': 5886, 'num_object_clones': 0, 'num_object_copies': 23544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6628, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3575}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:14.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:14.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6813", 'reported_seq': 6888, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:12.308639+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:12.308639+0000', 'last_peered': '2024-09-11T05:00:12.308639+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:12.308639+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6813, 'log_dups_size': 0, 'ondisk_log_size': 6813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48534688, 'num_objects': 5925, 'num_object_clones': 0, 'num_object_copies': 23700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6811, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:14.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:14.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6473", 'reported_seq': 6564, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:09.810017+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:09.810017+0000', 'last_peered': '2024-09-11T05:00:09.810017+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:09.810017+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6473, 'log_dups_size': 0, 'ondisk_log_size': 6473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46586472, 'num_objects': 5688, 'num_object_clones': 0, 'num_object_copies': 22752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2216, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5688, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6472, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3473}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:14.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:14.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6640", 'reported_seq': 6716, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:12.426902+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:12.426902+0000', 'last_peered': '2024-09-11T05:00:12.426902+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:12.426902+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6640, 'log_dups_size': 0, 'ondisk_log_size': 6640, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47314080, 'num_objects': 5776, 'num_object_clones': 0, 'num_object_copies': 23104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6638, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6624", 'reported_seq': 6693, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:08.987191+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:08.987191+0000', 'last_peered': '2024-09-11T05:00:08.987191+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:08.987191+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6624, 'log_dups_size': 0, 'ondisk_log_size': 6624, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48387232, 'num_objects': 5907, 'num_object_clones': 0, 'num_object_copies': 23628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6622, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3662}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6703", 'reported_seq': 6755, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:08.877107+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:08.877107+0000', 'last_peered': '2024-09-11T05:00:08.877107+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:08.877107+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6703, 'log_dups_size': 0, 'ondisk_log_size': 6703, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48635904, 'num_objects': 5937, 'num_object_clones': 0, 'num_object_copies': 23748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2318, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6703, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3619}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6767", 'reported_seq': 6837, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:12.385534+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:12.385534+0000', 'last_peered': '2024-09-11T05:00:12.385534+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:12.385534+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6767, 'log_dups_size': 0, 'ondisk_log_size': 6767, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48845984, 'num_objects': 5963, 'num_object_clones': 0, 'num_object_copies': 23852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2318, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6765, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3647}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6899", 'reported_seq': 7764, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:12.502866+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:12.502866+0000', 'last_peered': '2024-09-11T05:00:12.502866+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:12.502866+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6899, 'log_dups_size': 0, 'ondisk_log_size': 6899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48542880, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1962, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6897, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 401, 'num_bytes_recovered': 3252224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3964}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6792", 'reported_seq': 6865, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:08.920984+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:08.920984+0000', 'last_peered': '2024-09-11T05:00:08.920984+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:08.920984+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6792, 'log_dups_size': 0, 'ondisk_log_size': 6792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49113408, 'num_objects': 5996, 'num_object_clones': 0, 'num_object_copies': 23984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2391, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6789, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3608}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6670", 'reported_seq': 6758, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:09.859032+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:09.859032+0000', 'last_peered': '2024-09-11T05:00:09.859032+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:09.859032+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6670, 'log_dups_size': 0, 'ondisk_log_size': 6670, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47985824, 'num_objects': 5858, 'num_object_clones': 0, 'num_object_copies': 23432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6668, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3543}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6741", 'reported_seq': 6808, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:08.929814+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:08.929814+0000', 'last_peered': '2024-09-11T05:00:08.929814+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:08.929814+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6741, 'log_dups_size': 0, 'ondisk_log_size': 6741, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48426736, 'num_objects': 5912, 'num_object_clones': 0, 'num_object_copies': 23648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2323, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6738, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3592}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6785", 'reported_seq': 6864, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:09.817510+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:09.817510+0000', 'last_peered': '2024-09-11T05:00:09.817510+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:09.817510+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6785, 'log_dups_size': 0, 'ondisk_log_size': 6785, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48642640, 'num_objects': 5938, 'num_object_clones': 0, 'num_object_copies': 23752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2333, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6784, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3606}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6687", 'reported_seq': 6742, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:08.942343+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:08.942343+0000', 'last_peered': '2024-09-11T05:00:08.942343+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:08.942343+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6687, 'log_dups_size': 0, 'ondisk_log_size': 6687, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48370848, 'num_objects': 5905, 'num_object_clones': 0, 'num_object_copies': 23620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6685, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3618}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:14.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:14.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6591", 'reported_seq': 6688, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:09.834178+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:09.834178+0000', 'last_peered': '2024-09-11T05:00:09.834178+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:09.834178+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6591, 'log_dups_size': 0, 'ondisk_log_size': 6591, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47658144, 'num_objects': 5818, 'num_object_clones': 0, 'num_object_copies': 23272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6589, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:14.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:14.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6830", 'reported_seq': 6900, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:12.438418+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:12.438418+0000', 'last_peered': '2024-09-11T05:00:12.438418+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:12.438418+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6830, 'log_dups_size': 0, 'ondisk_log_size': 6830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48714912, 'num_objects': 5947, 'num_object_clones': 0, 'num_object_copies': 23788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6828, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3637}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:14.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:15.626 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:15.626 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34203/376536 objects degraded (9.084%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":342,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94134,"data_bytes":771098248,"bytes_used":2872979456,"bytes_avail":383674077184,"bytes_total":386547056640,"degraded_objects":34203,"degraded_total":376536,"degraded_ratio":0.090835936006118931,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":95530,"recovering_keys_per_sec":0,"num_objects_recovered":169,"num_bytes_recovered":1146880,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":246},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:15.653 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:15.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:16.303 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:16.328 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-11T05:00:15.712772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":764421768,"num_objects":93319,"num_object_clones":0,"num_object_copies":373276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101874,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108090,"ondisk_log_size":108090,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2787384,"kb_used_data":1385396,"kb_used_omap":14,"kb_used_meta":1401905,"kb_avail":374699976,"statfs":{"total":386547056640,"available":383692775424,"internally_reserved":0,"allocated":1418645504,"data_stored":1417925628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1435551286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,3,14,13,10,0,2,0,0,0,2,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27000832,"num_objects":-3296,"num_object_clones":0,"num_object_copies":-13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":712704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006231"},"pg_stats":[{"pgid":"1.f","version":"33'6718","reported_seq":6767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.007224+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:14.007224+0000","last_peered":"2024-09-11T05:00:14.007224+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:14.007224+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47480832,"num_objects":5796,"num_object_clones":0,"num_object_copies":23184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6813","reported_seq":6888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.308639+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:12.308639+0000","last_peered":"2024-09-11T05:00:12.308639+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:12.308639+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6813,"log_dups_size":0,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48534688,"num_objects":5925,"num_object_clones":0,"num_object_copies":23700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6543","reported_seq":6636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.895463+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:14.895463+0000","last_peered":"2024-09-11T05:00:14.895463+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:14.895463+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46013032,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6542,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6640","reported_seq":6716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.426902+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:12.426902+0000","last_peered":"2024-09-11T05:00:12.426902+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:12.426902+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6640,"log_dups_size":0,"ondisk_log_size":6640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314080,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6707","reported_seq":6776,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.072408+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:14.072408+0000","last_peered":"2024-09-11T05:00:14.072408+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:14.072408+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47707296,"num_objects":5824,"num_object_clones":0,"num_object_copies":23296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6705,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6804","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.063706+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:14.063706+0000","last_peered":"2024-09-11T05:00:14.063706+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:14.063706+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6804,"log_dups_size":0,"ondisk_log_size":6804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47808512,"num_objects":5836,"num_object_clones":0,"num_object_copies":23344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6804,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6767","reported_seq":6837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.385534+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:12.385534+0000","last_peered":"2024-09-11T05:00:12.385534+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:12.385534+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6767,"log_dups_size":0,"ondisk_log_size":6767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48845984,"num_objects":5963,"num_object_clones":0,"num_object_copies":23852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3647}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6899","reported_seq":7764,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.502866+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:12.502866+0000","last_peered":"2024-09-11T05:00:12.502866+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:12.502866+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48542880,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":401,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3964}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6874","reported_seq":6951,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.008781+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:14.008781+0000","last_peered":"2024-09-11T05:00:14.008781+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:14.008781+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6874,"log_dups_size":0,"ondisk_log_size":6874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48441664,"num_objects":5914,"num_object_clones":0,"num_object_copies":23656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6871,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6736","reported_seq":6828,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:15.052670+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:15.052670+0000","last_peered":"2024-09-11T05:00:15.052670+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:15.052670+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47445152,"num_objects":5792,"num_object_clones":0,"num_object_copies":23168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3479}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6806","reported_seq":6875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:13.972771+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:13.972771+0000","last_peered":"2024-09-11T05:00:13.972771+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:13.972771+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6806,"log_dups_size":0,"ondisk_log_size":6806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47894256,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6803,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6632","reported_seq":11227,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:13.483776+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:13.483776+0000","last_peered":"2024-09-11T05:00:13.483776+0000","last_clean":"2024-09-11T05:00:13.483776+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:13.483776+0000","last_undegraded":"2024-09-11T05:00:13.483776+0000","last_fullsized":"2024-09-11T05:00:13.483776+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47120384,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":442,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6873","reported_seq":6954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.864212+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:14.864212+0000","last_peered":"2024-09-11T05:00:14.864212+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:14.864212+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47921744,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6764","reported_seq":6821,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:13.765079+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:13.765079+0000","last_peered":"2024-09-11T05:00:13.765079+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:13.765079+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6764,"log_dups_size":0,"ondisk_log_size":6764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47740064,"num_objects":5828,"num_object_clones":0,"num_object_copies":23312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6762,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3542}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6684","reported_seq":6781,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.941776+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:14.941776+0000","last_peered":"2024-09-11T05:00:14.941776+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:14.941776+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46896288,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6830","reported_seq":6900,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:12.438418+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:12.438418+0000","last_peered":"2024-09-11T05:00:12.438418+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:12.438418+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48714912,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":764421768,"num_objects":93319,"num_object_clones":0,"num_object_copies":373276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101874,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1415311360,"data_stored":1415311360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108090,"ondisk_log_size":108090,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744304,"kb_used_data":373856,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93627536,"statfs":{"total":96636764160,"available":95874596864,"internally_reserved":0,"allocated":382828544,"data_stored":382648575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":379317968},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,3,3,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755500,"kb_used_data":373084,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93616340,"statfs":{"total":96636764160,"available":95863132160,"internally_reserved":0,"allocated":382038016,"data_stored":381858047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,7,6,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533288,"kb_used_data":262872,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93838552,"statfs":{"total":96636764160,"available":96090677248,"internally_reserved":0,"allocated":269180928,"data_stored":269000959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276885202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754292,"kb_used_data":375584,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93617548,"statfs":{"total":96636764160,"available":95864369152,"internally_reserved":0,"allocated":384598016,"data_stored":384418047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":387772111},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2,1,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383578112,"data_stored":383578112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268886016,"data_stored":268886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380952576,"data_stored":380952576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":381894656,"data_stored":381894656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:19.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:19.965 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:19.966 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:19.989 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-11T05:00:19.714595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":756901512,"num_objects":92401,"num_object_clones":0,"num_object_copies":369604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92401,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102792,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109008,"ondisk_log_size":109008,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825772,"kb_used_data":1375196,"kb_used_omap":14,"kb_used_meta":1450481,"kb_avail":374661588,"statfs":{"total":386547056640,"available":383653466112,"internally_reserved":0,"allocated":1408200704,"data_stored":1407480828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1485293110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,2,21,11,9,3,2,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24928256,"num_objects":-3043,"num_object_clones":0,"num_object_copies":-12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006161"},"pg_stats":[{"pgid":"1.f","version":"33'6808","reported_seq":6857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.033848+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:19.033848+0000","last_peered":"2024-09-11T05:00:19.033848+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:19.033848+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46743552,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3395}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6873","reported_seq":6947,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.516384+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:17.516384+0000","last_peered":"2024-09-11T05:00:17.516384+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:17.516384+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48043168,"num_objects":5865,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6871,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6543","reported_seq":6636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.895463+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:14.895463+0000","last_peered":"2024-09-11T05:00:14.895463+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:14.895463+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46013032,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6542,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6709","reported_seq":6784,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.437942+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:17.437942+0000","last_peered":"2024-09-11T05:00:17.437942+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:17.437942+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46748832,"num_objects":5707,"num_object_clones":0,"num_object_copies":22828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6779","reported_seq":6846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.094043+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:19.094043+0000","last_peered":"2024-09-11T05:00:19.094043+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:19.094043+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6779,"log_dups_size":0,"ondisk_log_size":6779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6886","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.045763+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:19.045763+0000","last_peered":"2024-09-11T05:00:19.045763+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:19.045763+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47136768,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6886,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6850","reported_seq":6920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.292194+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:17.292194+0000","last_peered":"2024-09-11T05:00:17.292194+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:17.292194+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48166048,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6962","reported_seq":7920,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.476971+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:17.476971+0000","last_peered":"2024-09-11T05:00:17.476971+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:17.476971+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6962,"log_dups_size":0,"ondisk_log_size":6962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48026784,"num_objects":5863,"num_object_clones":0,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3947}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6966","reported_seq":7043,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:18.985520+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:18.985520+0000","last_peered":"2024-09-11T05:00:18.985520+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:18.985520+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47688000,"num_objects":5822,"num_object_clones":0,"num_object_copies":23288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6736","reported_seq":6828,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:15.052670+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:15.052670+0000","last_peered":"2024-09-11T05:00:15.052670+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:15.052670+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47445152,"num_objects":5792,"num_object_clones":0,"num_object_copies":23168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3479}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6882","reported_seq":6951,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.041175+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:19.041175+0000","last_peered":"2024-09-11T05:00:19.041175+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:19.041175+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47271664,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6879,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6705","reported_seq":11300,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:18.474519+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:18.474519+0000","last_peered":"2024-09-11T05:00:18.474519+0000","last_clean":"2024-09-11T05:00:18.474519+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:18.474519+0000","last_undegraded":"2024-09-11T05:00:18.474519+0000","last_fullsized":"2024-09-11T05:00:18.474519+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46522368,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6873","reported_seq":6954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.864212+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:14.864212+0000","last_peered":"2024-09-11T05:00:14.864212+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:14.864212+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47921744,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6838","reported_seq":6895,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.086644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:19.086644+0000","last_peered":"2024-09-11T05:00:19.086644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:19.086644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6838,"log_dups_size":0,"ondisk_log_size":6838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47133856,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6836,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6684","reported_seq":6781,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.941776+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:14.941776+0000","last_peered":"2024-09-11T05:00:14.941776+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:14.941776+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46896288,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6914","reported_seq":6984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.214906+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:17.214906+0000","last_peered":"2024-09-11T05:00:17.214906+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:17.214906+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48026784,"num_objects":5863,"num_object_clones":0,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":756901512,"num_objects":92401,"num_object_clones":0,"num_object_copies":369604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92401,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102792,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1404743680,"data_stored":1404743680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109008,"ondisk_log_size":109008,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761636,"kb_used_data":368972,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93610204,"statfs":{"total":96636764160,"available":95856848896,"internally_reserved":0,"allocated":377827328,"data_stored":377647359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,9,3,1,3,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755500,"kb_used_data":373084,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93616340,"statfs":{"total":96636764160,"available":95863132160,"internally_reserved":0,"allocated":382038016,"data_stored":381858047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,7,6,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":537180,"kb_used_data":262604,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93834660,"statfs":{"total":96636764160,"available":96086691840,"internally_reserved":0,"allocated":268906496,"data_stored":268726527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281145042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771456,"kb_used_data":370536,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93600384,"statfs":{"total":96636764160,"available":95846793216,"internally_reserved":0,"allocated":379428864,"data_stored":379248895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,5,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378449920,"data_stored":378449920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268623872,"data_stored":268623872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380952576,"data_stored":380952576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376717312,"data_stored":376717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:19.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:20.608 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-11T05:00:19.714595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":756901512,"num_objects":92401,"num_object_clones":0,"num_object_copies":369604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92401,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102792,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109008,"ondisk_log_size":109008,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825772,"kb_used_data":1375196,"kb_used_omap":14,"kb_used_meta":1450481,"kb_avail":374661588,"statfs":{"total":386547056640,"available":383653466112,"internally_reserved":0,"allocated":1408200704,"data_stored":1407480828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1485293110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,2,21,11,9,3,2,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24928256,"num_objects":-3043,"num_object_clones":0,"num_object_copies":-12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006161"},"pg_stats":[{"pgid":"1.f","version":"33'6808","reported_seq":6857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.033848+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:19.033848+0000","last_peered":"2024-09-11T05:00:19.033848+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:19.033848+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46743552,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3395}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6873","reported_seq":6947,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.516384+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:17.516384+0000","last_peered":"2024-09-11T05:00:17.516384+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:17.516384+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48043168,"num_objects":5865,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6871,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6543","reported_seq":6636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.895463+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:14.895463+0000","last_peered":"2024-09-11T05:00:14.895463+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:14.895463+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46013032,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6542,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6709","reported_seq":6784,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.437942+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:17.437942+0000","last_peered":"2024-09-11T05:00:17.437942+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:17.437942+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46748832,"num_objects":5707,"num_object_clones":0,"num_object_copies":22828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6779","reported_seq":6846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.094043+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:19.094043+0000","last_peered":"2024-09-11T05:00:19.094043+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:19.094043+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6779,"log_dups_size":0,"ondisk_log_size":6779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6886","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.045763+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:19.045763+0000","last_peered":"2024-09-11T05:00:19.045763+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:19.045763+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47136768,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6886,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6850","reported_seq":6920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.292194+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:17.292194+0000","last_peered":"2024-09-11T05:00:17.292194+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:17.292194+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48166048,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6962","reported_seq":7920,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.476971+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:17.476971+0000","last_peered":"2024-09-11T05:00:17.476971+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:17.476971+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6962,"log_dups_size":0,"ondisk_log_size":6962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48026784,"num_objects":5863,"num_object_clones":0,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3947}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6966","reported_seq":7043,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:18.985520+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:18.985520+0000","last_peered":"2024-09-11T05:00:18.985520+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:18.985520+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47688000,"num_objects":5822,"num_object_clones":0,"num_object_copies":23288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6736","reported_seq":6828,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:15.052670+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:15.052670+0000","last_peered":"2024-09-11T05:00:15.052670+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:15.052670+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47445152,"num_objects":5792,"num_object_clones":0,"num_object_copies":23168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3479}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6882","reported_seq":6951,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.041175+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:19.041175+0000","last_peered":"2024-09-11T05:00:19.041175+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:19.041175+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47271664,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6879,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6705","reported_seq":11300,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:18.474519+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:18.474519+0000","last_peered":"2024-09-11T05:00:18.474519+0000","last_clean":"2024-09-11T05:00:18.474519+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:18.474519+0000","last_undegraded":"2024-09-11T05:00:18.474519+0000","last_fullsized":"2024-09-11T05:00:18.474519+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46522368,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6873","reported_seq":6954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.864212+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:14.864212+0000","last_peered":"2024-09-11T05:00:14.864212+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:14.864212+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47921744,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6838","reported_seq":6895,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.086644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:19.086644+0000","last_peered":"2024-09-11T05:00:19.086644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:19.086644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6838,"log_dups_size":0,"ondisk_log_size":6838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47133856,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6836,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6684","reported_seq":6781,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:14.941776+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:14.941776+0000","last_peered":"2024-09-11T05:00:14.941776+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:14.941776+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46896288,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3414}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6914","reported_seq":6984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.214906+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:17.214906+0000","last_peered":"2024-09-11T05:00:17.214906+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:17.214906+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48026784,"num_objects":5863,"num_object_clones":0,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":756901512,"num_objects":92401,"num_object_clones":0,"num_object_copies":369604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92401,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102792,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1404743680,"data_stored":1404743680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109008,"ondisk_log_size":109008,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761636,"kb_used_data":368972,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93610204,"statfs":{"total":96636764160,"available":95856848896,"internally_reserved":0,"allocated":377827328,"data_stored":377647359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,9,3,1,3,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755500,"kb_used_data":373084,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93616340,"statfs":{"total":96636764160,"available":95863132160,"internally_reserved":0,"allocated":382038016,"data_stored":381858047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,7,6,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":537180,"kb_used_data":262604,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93834660,"statfs":{"total":96636764160,"available":96086691840,"internally_reserved":0,"allocated":268906496,"data_stored":268726527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281145042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771456,"kb_used_data":370536,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93600384,"statfs":{"total":96636764160,"available":95846793216,"internally_reserved":0,"allocated":379428864,"data_stored":379248895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,5,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378449920,"data_stored":378449920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268623872,"data_stored":268623872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380952576,"data_stored":380952576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376717312,"data_stored":376717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:20.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:20.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6808", 'reported_seq': 6857, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:19.033848+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:19.033848+0000', 'last_peered': '2024-09-11T05:00:19.033848+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:19.033848+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6808, 'log_dups_size': 0, 'ondisk_log_size': 6808, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46743552, 'num_objects': 5706, 'num_object_clones': 0, 'num_object_copies': 22824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6808, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3395}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:20.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:20.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6873", 'reported_seq': 6947, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:17.516384+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:17.516384+0000', 'last_peered': '2024-09-11T05:00:17.516384+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:17.516384+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6873, 'log_dups_size': 0, 'ondisk_log_size': 6873, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48043168, 'num_objects': 5865, 'num_object_clones': 0, 'num_object_copies': 23460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6871, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:20.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6543", 'reported_seq': 6636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:14.895463+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:14.895463+0000', 'last_peered': '2024-09-11T05:00:14.895463+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:14.895463+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6543, 'log_dups_size': 0, 'ondisk_log_size': 6543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46013032, 'num_objects': 5618, 'num_object_clones': 0, 'num_object_copies': 22472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5618, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6542, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3404}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6709", 'reported_seq': 6784, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:17.437942+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:17.437942+0000', 'last_peered': '2024-09-11T05:00:17.437942+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:17.437942+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6709, 'log_dups_size': 0, 'ondisk_log_size': 6709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46748832, 'num_objects': 5707, 'num_object_clones': 0, 'num_object_copies': 22828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6707, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3433}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6779", 'reported_seq': 6846, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:19.094043+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:19.094043+0000', 'last_peered': '2024-09-11T05:00:19.094043+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:19.094043+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6779, 'log_dups_size': 0, 'ondisk_log_size': 6779, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47117472, 'num_objects': 5752, 'num_object_clones': 0, 'num_object_copies': 23008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6777, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3507}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6886", 'reported_seq': 6944, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:19.045763+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:19.045763+0000', 'last_peered': '2024-09-11T05:00:19.045763+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:19.045763+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6886, 'log_dups_size': 0, 'ondisk_log_size': 6886, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47136768, 'num_objects': 5754, 'num_object_clones': 0, 'num_object_copies': 23016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6886, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3439}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6850", 'reported_seq': 6920, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:17.292194+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:17.292194+0000', 'last_peered': '2024-09-11T05:00:17.292194+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:17.292194+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6850, 'log_dups_size': 0, 'ondisk_log_size': 6850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48166048, 'num_objects': 5880, 'num_object_clones': 0, 'num_object_copies': 23520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2318, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6848, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3564}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:20.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6962", 'reported_seq': 7920, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:17.476971+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:17.476971+0000', 'last_peered': '2024-09-11T05:00:17.476971+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:17.476971+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6962, 'log_dups_size': 0, 'ondisk_log_size': 6962, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48026784, 'num_objects': 5863, 'num_object_clones': 0, 'num_object_copies': 23452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6960, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 447, 'num_bytes_recovered': 3629056, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3947}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6966", 'reported_seq': 7043, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:18.985520+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:18.985520+0000', 'last_peered': '2024-09-11T05:00:18.985520+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:18.985520+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6966, 'log_dups_size': 0, 'ondisk_log_size': 6966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47688000, 'num_objects': 5822, 'num_object_clones': 0, 'num_object_copies': 23288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2389, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6963, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3436}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6736", 'reported_seq': 6828, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:15.052670+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:15.052670+0000', 'last_peered': '2024-09-11T05:00:15.052670+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:15.052670+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6736, 'log_dups_size': 0, 'ondisk_log_size': 6736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47445152, 'num_objects': 5792, 'num_object_clones': 0, 'num_object_copies': 23168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6734, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3479}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6882", 'reported_seq': 6951, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:19.041175+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:19.041175+0000', 'last_peered': '2024-09-11T05:00:19.041175+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:19.041175+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6882, 'log_dups_size': 0, 'ondisk_log_size': 6882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47271664, 'num_objects': 5771, 'num_object_clones': 0, 'num_object_copies': 23084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6879, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3452}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6873", 'reported_seq': 6954, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:14.864212+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:14.864212+0000', 'last_peered': '2024-09-11T05:00:14.864212+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:14.864212+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6873, 'log_dups_size': 0, 'ondisk_log_size': 6873, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47921744, 'num_objects': 5850, 'num_object_clones': 0, 'num_object_copies': 23400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2332, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6872, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:20.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6838", 'reported_seq': 6895, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:19.086644+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:19.086644+0000', 'last_peered': '2024-09-11T05:00:19.086644+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:19.086644+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6838, 'log_dups_size': 0, 'ondisk_log_size': 6838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47133856, 'num_objects': 5754, 'num_object_clones': 0, 'num_object_copies': 23016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2288, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6836, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3468}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:20.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:20.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6684", 'reported_seq': 6781, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:14.941776+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:14.941776+0000', 'last_peered': '2024-09-11T05:00:14.941776+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:14.941776+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6684, 'log_dups_size': 0, 'ondisk_log_size': 6684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46896288, 'num_objects': 5725, 'num_object_clones': 0, 'num_object_copies': 22900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6682, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:20.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:20.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6914", 'reported_seq': 6984, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:17.214906+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:17.214906+0000', 'last_peered': '2024-09-11T05:00:17.214906+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:17.214906+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6914, 'log_dups_size': 0, 'ondisk_log_size': 6914, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48026784, 'num_objects': 5863, 'num_object_clones': 0, 'num_object_copies': 23452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6912, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:20.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:21.423 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34146/369604 objects degraded (9.239%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":348,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":92401,"data_bytes":756901512,"bytes_used":2893590528,"bytes_avail":383653466112,"bytes_total":386547056640,"degraded_objects":34146,"degraded_total":369604,"degraded_ratio":0.092385363794764128,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":64820,"recovering_keys_per_sec":0,"num_objects_recovered":127,"num_bytes_recovered":778240,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:21.448 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:21.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:22.058 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:22.088 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-11T05:00:21.715500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":754075272,"num_objects":92056,"num_object_clones":0,"num_object_copies":368224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92056,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103137,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109353,"ondisk_log_size":109353,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2826256,"kb_used_data":1370112,"kb_used_omap":14,"kb_used_meta":1456049,"kb_avail":374661104,"statfs":{"total":386547056640,"available":383652970496,"internally_reserved":0,"allocated":1402994688,"data_stored":1402274812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1490994742},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8,4,19,7,3,6,0,0,0,0,0,2,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23166976,"num_objects":-2828,"num_object_clones":0,"num_object_copies":-11312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2828,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006197"},"pg_stats":[{"pgid":"1.f","version":"33'6808","reported_seq":6857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.033848+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:19.033848+0000","last_peered":"2024-09-11T05:00:19.033848+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:19.033848+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46743552,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3395}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6873","reported_seq":6947,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.516384+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:17.516384+0000","last_peered":"2024-09-11T05:00:17.516384+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:17.516384+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48043168,"num_objects":5865,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6871,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6635","reported_seq":6728,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:20.060298+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:20.060298+0000","last_peered":"2024-09-11T05:00:20.060298+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:20.060298+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45259368,"num_objects":5526,"num_object_clones":0,"num_object_copies":22104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6634,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3312}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6709","reported_seq":6784,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.437942+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:17.437942+0000","last_peered":"2024-09-11T05:00:17.437942+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:17.437942+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46748832,"num_objects":5707,"num_object_clones":0,"num_object_copies":22828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6779","reported_seq":6846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.094043+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:19.094043+0000","last_peered":"2024-09-11T05:00:19.094043+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:19.094043+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6779,"log_dups_size":0,"ondisk_log_size":6779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47117472,"num_objects":5752,"num_object_clones":0,"num_object_copies":23008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6886","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.045763+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:19.045763+0000","last_peered":"2024-09-11T05:00:19.045763+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:19.045763+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47136768,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6886,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6850","reported_seq":6920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.292194+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:17.292194+0000","last_peered":"2024-09-11T05:00:17.292194+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:17.292194+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48166048,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6962","reported_seq":7920,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.476971+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:17.476971+0000","last_peered":"2024-09-11T05:00:17.476971+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:17.476971+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6962,"log_dups_size":0,"ondisk_log_size":6962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48026784,"num_objects":5863,"num_object_clones":0,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3947}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6966","reported_seq":7043,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:18.985520+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:18.985520+0000","last_peered":"2024-09-11T05:00:18.985520+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:18.985520+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47688000,"num_objects":5822,"num_object_clones":0,"num_object_copies":23288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6812","reported_seq":6904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:20.110798+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:20.110798+0000","last_peered":"2024-09-11T05:00:20.110798+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:20.110798+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822560,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6810,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3403}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6882","reported_seq":6951,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.041175+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:19.041175+0000","last_peered":"2024-09-11T05:00:19.041175+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:19.041175+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47271664,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6879,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6705","reported_seq":11300,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:18.474519+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:18.474519+0000","last_peered":"2024-09-11T05:00:18.474519+0000","last_clean":"2024-09-11T05:00:18.474519+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:18.474519+0000","last_undegraded":"2024-09-11T05:00:18.474519+0000","last_fullsized":"2024-09-11T05:00:18.474519+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46522368,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6953","reported_seq":7034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:20.105888+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:20.105888+0000","last_peered":"2024-09-11T05:00:20.105888+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:20.105888+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6953,"log_dups_size":0,"ondisk_log_size":6953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47266384,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6952,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3439}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6838","reported_seq":6895,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.086644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:19.086644+0000","last_peered":"2024-09-11T05:00:19.086644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:19.086644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6838,"log_dups_size":0,"ondisk_log_size":6838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47133856,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6836,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3468}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6781","reported_seq":6878,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:19.912522+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:19.912522+0000","last_peered":"2024-09-11T05:00:19.912522+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:19.912522+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46101664,"num_objects":5628,"num_object_clones":0,"num_object_copies":22512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3317}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6914","reported_seq":6984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:17.214906+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:17.214906+0000","last_peered":"2024-09-11T05:00:17.214906+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:17.214906+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48026784,"num_objects":5863,"num_object_clones":0,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":754075272,"num_objects":92056,"num_object_clones":0,"num_object_copies":368224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92056,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103137,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1399517184,"data_stored":1399517184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109353,"ondisk_log_size":109353,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761636,"kb_used_data":368972,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93610204,"statfs":{"total":96636764160,"available":95856848896,"internally_reserved":0,"allocated":377827328,"data_stored":377647359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,9,3,1,3,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755984,"kb_used_data":368000,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93615856,"statfs":{"total":96636764160,"available":95862636544,"internally_reserved":0,"allocated":376832000,"data_stored":376652031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,0,3,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":537180,"kb_used_data":262604,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93834660,"statfs":{"total":96636764160,"available":96086691840,"internally_reserved":0,"allocated":268906496,"data_stored":268726527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281145042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771456,"kb_used_data":370536,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93600384,"statfs":{"total":96636764160,"available":95846793216,"internally_reserved":0,"allocated":379428864,"data_stored":379248895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,5,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378449920,"data_stored":378449920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268623872,"data_stored":268623872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375726080,"data_stored":375726080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376717312,"data_stored":376717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:25.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:25.735 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:25.736 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:25.767 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-11T05:00:25.717241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":744359560,"num_objects":90870,"num_object_clones":0,"num_object_copies":363480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104323,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110539,"ondisk_log_size":110539,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807464,"kb_used_data":1355424,"kb_used_omap":14,"kb_used_meta":1451889,"kb_avail":374679896,"statfs":{"total":386547056640,"available":383672213504,"internally_reserved":0,"allocated":1387954176,"data_stored":1387234300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1486734902},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,8,12,5,12,3,6,0,0,4,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26738688,"num_objects":-3264,"num_object_clones":0,"num_object_copies":-13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006308"},"pg_stats":[{"pgid":"1.f","version":"33'6887","reported_seq":6936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.070318+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:24.070318+0000","last_peered":"2024-09-11T05:00:24.070318+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:24.070318+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46096384,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6966","reported_seq":7043,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.493927+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:22.493927+0000","last_peered":"2024-09-11T05:00:22.493927+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:22.493927+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281312,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6964,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6710","reported_seq":6805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.035233+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:25.035233+0000","last_peered":"2024-09-11T05:00:25.035233+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:25.035233+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44644968,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6709,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6777","reported_seq":6853,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.517860+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:22.517860+0000","last_peered":"2024-09-11T05:00:22.517860+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:22.517860+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191776,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6850","reported_seq":6917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.136705+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:24.136705+0000","last_peered":"2024-09-11T05:00:24.136705+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:24.136705+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46535840,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6974","reported_seq":7032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:23.898244+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:23.898244+0000","last_peered":"2024-09-11T05:00:23.898244+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:23.898244+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6974,"log_dups_size":0,"ondisk_log_size":6974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46415872,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6974,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6913","reported_seq":6985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.309701+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:22.309701+0000","last_peered":"2024-09-11T05:00:22.309701+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:22.309701+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6913,"log_dups_size":0,"ondisk_log_size":6913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47649952,"num_objects":5817,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6911,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3502}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7039","reported_seq":8095,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.474604+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:22.474604+0000","last_peered":"2024-09-11T05:00:22.474604+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:22.474604+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396000,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7052","reported_seq":7128,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.131888+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:24.131888+0000","last_peered":"2024-09-11T05:00:24.131888+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:24.131888+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7052,"log_dups_size":0,"ondisk_log_size":7052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46983488,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7049,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6892","reported_seq":6986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.035404+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:25.035404+0000","last_peered":"2024-09-11T05:00:25.035404+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:25.035404+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6892,"log_dups_size":0,"ondisk_log_size":6892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46167200,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6890,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3324}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6950","reported_seq":7019,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.050112+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:24.050112+0000","last_peered":"2024-09-11T05:00:24.050112+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:24.050112+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46714608,"num_objects":5703,"num_object_clones":0,"num_object_copies":22812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6766","reported_seq":11361,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:23.435861+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:23.435861+0000","last_peered":"2024-09-11T05:00:23.435861+0000","last_clean":"2024-09-11T05:00:23.435861+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:23.435861+0000","last_undegraded":"2024-09-11T05:00:23.435861+0000","last_fullsized":"2024-09-11T05:00:23.435861+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6766,"log_dups_size":0,"ondisk_log_size":6766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46022656,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":576,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7025","reported_seq":7106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.980459+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:24.980459+0000","last_peered":"2024-09-11T05:00:24.980459+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:24.980459+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46676560,"num_objects":5698,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7024,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6896","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.114980+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:24.114980+0000","last_peered":"2024-09-11T05:00:24.114980+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:24.114980+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46658720,"num_objects":5696,"num_object_clones":0,"num_object_copies":22784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6855","reported_seq":6958,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.036505+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:25.036505+0000","last_peered":"2024-09-11T05:00:25.036505+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:25.036505+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495456,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6987","reported_seq":7059,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.542126+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:22.542126+0000","last_peered":"2024-09-11T05:00:22.542126+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:22.542126+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47428768,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":744359560,"num_objects":90870,"num_object_clones":0,"num_object_copies":363480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104323,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384558592,"data_stored":1384558592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110539,"ondisk_log_size":110539,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761028,"kb_used_data":364204,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93610812,"statfs":{"total":96636764160,"available":95857471488,"internally_reserved":0,"allocated":372944896,"data_stored":372764927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1,3,1,4,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759608,"kb_used_data":363276,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93612232,"statfs":{"total":96636764160,"available":95858925568,"internally_reserved":0,"allocated":371994624,"data_stored":371814655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,4,1,1,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528620,"kb_used_data":262364,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93843220,"statfs":{"total":96636764160,"available":96095457280,"internally_reserved":0,"allocated":268660736,"data_stored":268480767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272625362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758208,"kb_used_data":365580,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93613632,"statfs":{"total":96636764160,"available":95860359168,"internally_reserved":0,"allocated":374353920,"data_stored":374173951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,0,0,1,1,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373493760,"data_stored":373493760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268390400,"data_stored":268390400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370831360,"data_stored":370831360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371843072,"data_stored":371843072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:25.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:26.375 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:26.377 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:26.401 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-11T05:00:25.717241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":744359560,"num_objects":90870,"num_object_clones":0,"num_object_copies":363480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104323,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110539,"ondisk_log_size":110539,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807464,"kb_used_data":1355424,"kb_used_omap":14,"kb_used_meta":1451889,"kb_avail":374679896,"statfs":{"total":386547056640,"available":383672213504,"internally_reserved":0,"allocated":1387954176,"data_stored":1387234300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1486734902},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,8,12,5,12,3,6,0,0,4,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26738688,"num_objects":-3264,"num_object_clones":0,"num_object_copies":-13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006308"},"pg_stats":[{"pgid":"1.f","version":"33'6887","reported_seq":6936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.070318+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:24.070318+0000","last_peered":"2024-09-11T05:00:24.070318+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:24.070318+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46096384,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6966","reported_seq":7043,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.493927+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:22.493927+0000","last_peered":"2024-09-11T05:00:22.493927+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:22.493927+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281312,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6964,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6710","reported_seq":6805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.035233+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:25.035233+0000","last_peered":"2024-09-11T05:00:25.035233+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:25.035233+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44644968,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6709,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6777","reported_seq":6853,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.517860+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:22.517860+0000","last_peered":"2024-09-11T05:00:22.517860+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:22.517860+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191776,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6850","reported_seq":6917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.136705+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:24.136705+0000","last_peered":"2024-09-11T05:00:24.136705+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:24.136705+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46535840,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6974","reported_seq":7032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:23.898244+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:23.898244+0000","last_peered":"2024-09-11T05:00:23.898244+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:23.898244+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6974,"log_dups_size":0,"ondisk_log_size":6974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46415872,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6974,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6913","reported_seq":6985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.309701+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:22.309701+0000","last_peered":"2024-09-11T05:00:22.309701+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:22.309701+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6913,"log_dups_size":0,"ondisk_log_size":6913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47649952,"num_objects":5817,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6911,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3502}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7039","reported_seq":8095,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.474604+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:22.474604+0000","last_peered":"2024-09-11T05:00:22.474604+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:22.474604+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396000,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7052","reported_seq":7128,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.131888+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:24.131888+0000","last_peered":"2024-09-11T05:00:24.131888+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:24.131888+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7052,"log_dups_size":0,"ondisk_log_size":7052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46983488,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7049,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6892","reported_seq":6986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.035404+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:25.035404+0000","last_peered":"2024-09-11T05:00:25.035404+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:25.035404+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6892,"log_dups_size":0,"ondisk_log_size":6892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46167200,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6890,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3324}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6950","reported_seq":7019,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.050112+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:24.050112+0000","last_peered":"2024-09-11T05:00:24.050112+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:24.050112+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46714608,"num_objects":5703,"num_object_clones":0,"num_object_copies":22812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6766","reported_seq":11361,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:23.435861+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:23.435861+0000","last_peered":"2024-09-11T05:00:23.435861+0000","last_clean":"2024-09-11T05:00:23.435861+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:23.435861+0000","last_undegraded":"2024-09-11T05:00:23.435861+0000","last_fullsized":"2024-09-11T05:00:23.435861+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6766,"log_dups_size":0,"ondisk_log_size":6766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46022656,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":576,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7025","reported_seq":7106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.980459+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:24.980459+0000","last_peered":"2024-09-11T05:00:24.980459+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:24.980459+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46676560,"num_objects":5698,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7024,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6896","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.114980+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:24.114980+0000","last_peered":"2024-09-11T05:00:24.114980+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:24.114980+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46658720,"num_objects":5696,"num_object_clones":0,"num_object_copies":22784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6855","reported_seq":6958,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.036505+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:25.036505+0000","last_peered":"2024-09-11T05:00:25.036505+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:25.036505+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495456,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6987","reported_seq":7059,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:22.542126+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:22.542126+0000","last_peered":"2024-09-11T05:00:22.542126+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:22.542126+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47428768,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":744359560,"num_objects":90870,"num_object_clones":0,"num_object_copies":363480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104323,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384558592,"data_stored":1384558592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110539,"ondisk_log_size":110539,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761028,"kb_used_data":364204,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93610812,"statfs":{"total":96636764160,"available":95857471488,"internally_reserved":0,"allocated":372944896,"data_stored":372764927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1,3,1,4,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759608,"kb_used_data":363276,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93612232,"statfs":{"total":96636764160,"available":95858925568,"internally_reserved":0,"allocated":371994624,"data_stored":371814655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,4,1,1,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528620,"kb_used_data":262364,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93843220,"statfs":{"total":96636764160,"available":96095457280,"internally_reserved":0,"allocated":268660736,"data_stored":268480767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272625362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758208,"kb_used_data":365580,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93613632,"statfs":{"total":96636764160,"available":95860359168,"internally_reserved":0,"allocated":374353920,"data_stored":374173951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,0,0,1,1,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373493760,"data_stored":373493760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268390400,"data_stored":268390400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370831360,"data_stored":370831360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371843072,"data_stored":371843072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:26.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6887", 'reported_seq': 6936, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:24.070318+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:24.070318+0000', 'last_peered': '2024-09-11T05:00:24.070318+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:24.070318+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6887, 'log_dups_size': 0, 'ondisk_log_size': 6887, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46096384, 'num_objects': 5627, 'num_object_clones': 0, 'num_object_copies': 22508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6887, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6966", 'reported_seq': 7043, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:22.493927+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:22.493927+0000', 'last_peered': '2024-09-11T05:00:22.493927+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:22.493927+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6966, 'log_dups_size': 0, 'ondisk_log_size': 6966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47281312, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 23088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6964, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6710", 'reported_seq': 6805, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:25.035233+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:25.035233+0000', 'last_peered': '2024-09-11T05:00:25.035233+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:25.035233+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6710, 'log_dups_size': 0, 'ondisk_log_size': 6710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44644968, 'num_objects': 5451, 'num_object_clones': 0, 'num_object_copies': 21804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5451, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6709, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3238}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6777", 'reported_seq': 6853, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:22.517860+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:22.517860+0000', 'last_peered': '2024-09-11T05:00:22.517860+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:22.517860+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6777, 'log_dups_size': 0, 'ondisk_log_size': 6777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46191776, 'num_objects': 5639, 'num_object_clones': 0, 'num_object_copies': 22556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6775, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3365}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:26.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6850", 'reported_seq': 6917, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:24.136705+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:24.136705+0000', 'last_peered': '2024-09-11T05:00:24.136705+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:24.136705+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6850, 'log_dups_size': 0, 'ondisk_log_size': 6850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46535840, 'num_objects': 5681, 'num_object_clones': 0, 'num_object_copies': 22724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6848, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3436}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6974", 'reported_seq': 7032, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:23.898244+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:23.898244+0000', 'last_peered': '2024-09-11T05:00:23.898244+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:23.898244+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6974, 'log_dups_size': 0, 'ondisk_log_size': 6974, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46415872, 'num_objects': 5666, 'num_object_clones': 0, 'num_object_copies': 22664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6974, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6913", 'reported_seq': 6985, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:22.309701+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:22.309701+0000', 'last_peered': '2024-09-11T05:00:22.309701+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:22.309701+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6913, 'log_dups_size': 0, 'ondisk_log_size': 6913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47649952, 'num_objects': 5817, 'num_object_clones': 0, 'num_object_copies': 23268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6911, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3502}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7039", 'reported_seq': 8095, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:22.474604+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:22.474604+0000', 'last_peered': '2024-09-11T05:00:22.474604+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:22.474604+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7039, 'log_dups_size': 0, 'ondisk_log_size': 7039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47396000, 'num_objects': 5786, 'num_object_clones': 0, 'num_object_copies': 23144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1867, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7037, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 496, 'num_bytes_recovered': 4022272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3919}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7052", 'reported_seq': 7128, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:24.131888+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:24.131888+0000', 'last_peered': '2024-09-11T05:00:24.131888+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:24.131888+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7052, 'log_dups_size': 0, 'ondisk_log_size': 7052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46983488, 'num_objects': 5736, 'num_object_clones': 0, 'num_object_copies': 22944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2389, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7049, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3350}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:26.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6892", 'reported_seq': 6986, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:25.035404+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:25.035404+0000', 'last_peered': '2024-09-11T05:00:25.035404+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:25.035404+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6892, 'log_dups_size': 0, 'ondisk_log_size': 6892, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46167200, 'num_objects': 5636, 'num_object_clones': 0, 'num_object_copies': 22544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6890, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3324}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6950", 'reported_seq': 7019, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:24.050112+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:24.050112+0000', 'last_peered': '2024-09-11T05:00:24.050112+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:24.050112+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6950, 'log_dups_size': 0, 'ondisk_log_size': 6950, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46714608, 'num_objects': 5703, 'num_object_clones': 0, 'num_object_copies': 22812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6947, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3384}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7025", 'reported_seq': 7106, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:24.980459+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:24.980459+0000', 'last_peered': '2024-09-11T05:00:24.980459+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:24.980459+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7025, 'log_dups_size': 0, 'ondisk_log_size': 7025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46676560, 'num_objects': 5698, 'num_object_clones': 0, 'num_object_copies': 22792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2331, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7024, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3368}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6896", 'reported_seq': 6953, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:24.114980+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:24.114980+0000', 'last_peered': '2024-09-11T05:00:24.114980+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:24.114980+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6896, 'log_dups_size': 0, 'ondisk_log_size': 6896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46658720, 'num_objects': 5696, 'num_object_clones': 0, 'num_object_copies': 22784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2288, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6894, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6855", 'reported_seq': 6958, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:25.036505+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:25.036505+0000', 'last_peered': '2024-09-11T05:00:25.036505+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:25.036505+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6855, 'log_dups_size': 0, 'ondisk_log_size': 6855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45495456, 'num_objects': 5554, 'num_object_clones': 0, 'num_object_copies': 22216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6853, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:26.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6987", 'reported_seq': 7059, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:22.542126+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:22.542126+0000', 'last_peered': '2024-09-11T05:00:22.542126+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:22.542126+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6987, 'log_dups_size': 0, 'ondisk_log_size': 6987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47428768, 'num_objects': 5790, 'num_object_clones': 0, 'num_object_copies': 23160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6985, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3481}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:26.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:27.251 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34088/363480 objects degraded (9.378%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":354,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":90870,"data_bytes":744359560,"bytes_used":2874843136,"bytes_avail":383672213504,"bytes_total":386547056640,"degraded_objects":34088,"degraded_total":363480,"degraded_ratio":0.093782326400352153,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":64136,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":770048,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":271},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:27.275 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:27.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:27.835 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:27.856 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-11T05:00:27.718254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":741344904,"num_objects":90502,"num_object_clones":0,"num_object_copies":362008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90502,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104691,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110907,"ondisk_log_size":110907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806956,"kb_used_data":1350792,"kb_used_omap":14,"kb_used_meta":1456049,"kb_avail":374680404,"statfs":{"total":386547056640,"available":383672733696,"internally_reserved":0,"allocated":1383211008,"data_stored":1382491132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1490994742},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,8,5,15,3,5,1,0,0,0,0,2,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23076864,"num_objects":-2817,"num_object_clones":0,"num_object_copies":-11268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005482"},"pg_stats":[{"pgid":"1.f","version":"33'6887","reported_seq":6936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.070318+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:24.070318+0000","last_peered":"2024-09-11T05:00:24.070318+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:24.070318+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46096384,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7038","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.347085+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:27.347085+0000","last_peered":"2024-09-11T05:00:27.347085+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:27.347085+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7038,"log_dups_size":0,"ondisk_log_size":7038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46691488,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7036,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6710","reported_seq":6805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.035233+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:25.035233+0000","last_peered":"2024-09-11T05:00:25.035233+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:25.035233+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44644968,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6709,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6840","reported_seq":6916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.500072+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:27.500072+0000","last_peered":"2024-09-11T05:00:27.500072+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:27.500072+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6840,"log_dups_size":0,"ondisk_log_size":6840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45675680,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6850","reported_seq":6917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.136705+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:24.136705+0000","last_peered":"2024-09-11T05:00:24.136705+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:24.136705+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46535840,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6974","reported_seq":7032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:23.898244+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:23.898244+0000","last_peered":"2024-09-11T05:00:23.898244+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:23.898244+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6974,"log_dups_size":0,"ondisk_log_size":6974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46415872,"num_objects":5666,"num_object_clones":0,"num_object_copies":22664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6974,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7005","reported_seq":7077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.496590+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:27.496590+0000","last_peered":"2024-09-11T05:00:27.496590+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:27.496590+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46896288,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7003,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3410}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7107","reported_seq":8271,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.577171+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:27.577171+0000","last_peered":"2024-09-11T05:00:27.577171+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:27.577171+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7107,"log_dups_size":0,"ondisk_log_size":7107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46838944,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":550,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7052","reported_seq":7128,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.131888+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:24.131888+0000","last_peered":"2024-09-11T05:00:24.131888+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:24.131888+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7052,"log_dups_size":0,"ondisk_log_size":7052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46983488,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7049,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6892","reported_seq":6986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.035404+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:25.035404+0000","last_peered":"2024-09-11T05:00:25.035404+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:25.035404+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6892,"log_dups_size":0,"ondisk_log_size":6892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46167200,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6890,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3324}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6950","reported_seq":7019,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.050112+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:24.050112+0000","last_peered":"2024-09-11T05:00:24.050112+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:24.050112+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46714608,"num_objects":5703,"num_object_clones":0,"num_object_copies":22812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6766","reported_seq":11361,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:23.435861+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:23.435861+0000","last_peered":"2024-09-11T05:00:23.435861+0000","last_clean":"2024-09-11T05:00:23.435861+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:23.435861+0000","last_undegraded":"2024-09-11T05:00:23.435861+0000","last_fullsized":"2024-09-11T05:00:23.435861+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6766,"log_dups_size":0,"ondisk_log_size":6766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46022656,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":576,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7025","reported_seq":7106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.980459+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:24.980459+0000","last_peered":"2024-09-11T05:00:24.980459+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:24.980459+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46676560,"num_objects":5698,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7024,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6896","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.114980+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:24.114980+0000","last_peered":"2024-09-11T05:00:24.114980+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:24.114980+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46658720,"num_objects":5696,"num_object_clones":0,"num_object_copies":22784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6855","reported_seq":6958,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.036505+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:25.036505+0000","last_peered":"2024-09-11T05:00:25.036505+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:25.036505+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495456,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7060","reported_seq":7132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.531269+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:27.531269+0000","last_peered":"2024-09-11T05:00:27.531269+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:27.531269+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830752,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":741344904,"num_objects":90502,"num_object_clones":0,"num_object_copies":362008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90502,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104691,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1379725312,"data_stored":1379725312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110907,"ondisk_log_size":110907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761028,"kb_used_data":364204,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93610812,"statfs":{"total":96636764160,"available":95857471488,"internally_reserved":0,"allocated":372944896,"data_stored":372764927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1,3,1,4,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759608,"kb_used_data":363276,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93612232,"statfs":{"total":96636764160,"available":95858925568,"internally_reserved":0,"allocated":371994624,"data_stored":371814655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,4,1,1,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528620,"kb_used_data":262364,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93843220,"statfs":{"total":96636764160,"available":96095457280,"internally_reserved":0,"allocated":268660736,"data_stored":268480767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":272625362},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757700,"kb_used_data":360948,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93614140,"statfs":{"total":96636764160,"available":95860879360,"internally_reserved":0,"allocated":369610752,"data_stored":369430783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,1,0,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368660480,"data_stored":368660480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268390400,"data_stored":268390400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370831360,"data_stored":370831360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371843072,"data_stored":371843072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:30.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:31.472 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:31.498 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-11T05:00:29.719136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":737322632,"num_objects":90011,"num_object_clones":0,"num_object_copies":360044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105182,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111398,"ondisk_log_size":111398,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2810332,"kb_used_data":1345856,"kb_used_omap":14,"kb_used_meta":1464369,"kb_avail":374677028,"statfs":{"total":386547056640,"available":383669276672,"internally_reserved":0,"allocated":1378156544,"data_stored":1377436668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1499514422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,15,3,12,4,1,1,0,0,0,0,2,2,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24158208,"num_objects":-2949,"num_object_clones":0,"num_object_copies":-11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2949,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005445"},"pg_stats":[{"pgid":"1.f","version":"33'6959","reported_seq":7008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.121209+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:29.121209+0000","last_peered":"2024-09-11T05:00:29.121209+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:29.121209+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45506560,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6959,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7038","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.347085+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:27.347085+0000","last_peered":"2024-09-11T05:00:27.347085+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:27.347085+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7038,"log_dups_size":0,"ondisk_log_size":7038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46691488,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7036,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6710","reported_seq":6805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.035233+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:25.035233+0000","last_peered":"2024-09-11T05:00:25.035233+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:25.035233+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44644968,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6709,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6840","reported_seq":6916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.500072+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:27.500072+0000","last_peered":"2024-09-11T05:00:27.500072+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:27.500072+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6840,"log_dups_size":0,"ondisk_log_size":6840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45675680,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6916","reported_seq":6985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.106514+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:29.106514+0000","last_peered":"2024-09-11T05:00:29.106514+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:29.106514+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45995168,"num_objects":5615,"num_object_clones":0,"num_object_copies":22460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6914,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7039","reported_seq":7097,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.138403+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:29.138403+0000","last_peered":"2024-09-11T05:00:29.138403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:29.138403+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45883392,"num_objects":5601,"num_object_clones":0,"num_object_copies":22404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7039,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7005","reported_seq":7077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.496590+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:27.496590+0000","last_peered":"2024-09-11T05:00:27.496590+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:27.496590+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46896288,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7003,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3410}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7107","reported_seq":8271,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.577171+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:27.577171+0000","last_peered":"2024-09-11T05:00:27.577171+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:27.577171+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7107,"log_dups_size":0,"ondisk_log_size":7107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46838944,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":550,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7116","reported_seq":7193,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.115196+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:29.115196+0000","last_peered":"2024-09-11T05:00:29.115196+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:29.115196+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46459200,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7113,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6892","reported_seq":6986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.035404+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:25.035404+0000","last_peered":"2024-09-11T05:00:25.035404+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:25.035404+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6892,"log_dups_size":0,"ondisk_log_size":6892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46167200,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6890,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3324}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7024","reported_seq":7093,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.161753+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:29.161753+0000","last_peered":"2024-09-11T05:00:29.161753+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:29.161753+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46108400,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7021,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6843","reported_seq":11438,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:28.588275+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:28.588275+0000","last_peered":"2024-09-11T05:00:28.588275+0000","last_clean":"2024-09-11T05:00:28.588275+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:28.588275+0000","last_undegraded":"2024-09-11T05:00:28.588275+0000","last_fullsized":"2024-09-11T05:00:28.588275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6843,"log_dups_size":0,"ondisk_log_size":6843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45391872,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":653,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7025","reported_seq":7106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:24.980459+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:24.980459+0000","last_peered":"2024-09-11T05:00:24.980459+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:24.980459+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46676560,"num_objects":5698,"num_object_clones":0,"num_object_copies":22792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7024,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6969","reported_seq":7026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.169700+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:29.169700+0000","last_peered":"2024-09-11T05:00:29.169700+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:29.169700+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6969,"log_dups_size":0,"ondisk_log_size":6969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6855","reported_seq":6958,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:25.036505+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:25.036505+0000","last_peered":"2024-09-11T05:00:25.036505+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:25.036505+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495456,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7060","reported_seq":7132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.531269+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:27.531269+0000","last_peered":"2024-09-11T05:00:27.531269+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:27.531269+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830752,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":737322632,"num_objects":90011,"num_object_clones":0,"num_object_copies":360044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105182,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1374605312,"data_stored":1374605312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111398,"ondisk_log_size":111398,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760500,"kb_used_data":359524,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93611340,"statfs":{"total":96636764160,"available":95858012160,"internally_reserved":0,"allocated":368152576,"data_stored":367972607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,2,5,2,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759608,"kb_used_data":363276,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93612232,"statfs":{"total":96636764160,"available":95858925568,"internally_reserved":0,"allocated":371994624,"data_stored":371814655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,4,1,1,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532524,"kb_used_data":262108,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93839316,"statfs":{"total":96636764160,"available":96091459584,"internally_reserved":0,"allocated":268398592,"data_stored":268218623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276885202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757700,"kb_used_data":360948,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93614140,"statfs":{"total":96636764160,"available":95860879360,"internally_reserved":0,"allocated":369610752,"data_stored":369430783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,1,0,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368660480,"data_stored":368660480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268103680,"data_stored":268103680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370831360,"data_stored":370831360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367009792,"data_stored":367009792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:31.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:32.134 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:32.134 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:32.159 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-11T05:00:31.720680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":734824072,"num_objects":89706,"num_object_clones":0,"num_object_copies":358824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89706,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105487,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111703,"ondisk_log_size":111703,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809688,"kb_used_data":1341064,"kb_used_omap":14,"kb_used_meta":1468529,"kb_avail":374677672,"statfs":{"total":386547056640,"available":383669936128,"internally_reserved":0,"allocated":1373249536,"data_stored":1372529660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1503774262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,14,3,13,4,0,1,0,0,0,0,2,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22077440,"num_objects":-2695,"num_object_clones":0,"num_object_copies":-10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006085"},"pg_stats":[{"pgid":"1.f","version":"33'6959","reported_seq":7008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.121209+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:29.121209+0000","last_peered":"2024-09-11T05:00:29.121209+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:29.121209+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45506560,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6959,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7038","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.347085+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:27.347085+0000","last_peered":"2024-09-11T05:00:27.347085+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:27.347085+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7038,"log_dups_size":0,"ondisk_log_size":7038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46691488,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7036,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6779","reported_seq":6874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:30.042336+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:30.042336+0000","last_peered":"2024-09-11T05:00:30.042336+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:30.042336+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6779,"log_dups_size":0,"ondisk_log_size":6779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079720,"num_objects":5382,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6778,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6840","reported_seq":6916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.500072+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:27.500072+0000","last_peered":"2024-09-11T05:00:27.500072+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:27.500072+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6840,"log_dups_size":0,"ondisk_log_size":6840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45675680,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6916","reported_seq":6985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.106514+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:29.106514+0000","last_peered":"2024-09-11T05:00:29.106514+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:29.106514+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45995168,"num_objects":5615,"num_object_clones":0,"num_object_copies":22460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6914,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7039","reported_seq":7097,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.138403+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:29.138403+0000","last_peered":"2024-09-11T05:00:29.138403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:29.138403+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45883392,"num_objects":5601,"num_object_clones":0,"num_object_copies":22404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7039,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7005","reported_seq":7077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.496590+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:27.496590+0000","last_peered":"2024-09-11T05:00:27.496590+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:27.496590+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46896288,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7003,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3410}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7107","reported_seq":8271,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.577171+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:27.577171+0000","last_peered":"2024-09-11T05:00:27.577171+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:27.577171+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7107,"log_dups_size":0,"ondisk_log_size":7107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46838944,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":550,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7116","reported_seq":7193,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.115196+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:29.115196+0000","last_peered":"2024-09-11T05:00:29.115196+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:29.115196+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46459200,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7113,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6978","reported_seq":7072,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.959390+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:29.959390+0000","last_peered":"2024-09-11T05:00:29.959390+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:29.959390+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45462688,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7024","reported_seq":7093,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.161753+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:29.161753+0000","last_peered":"2024-09-11T05:00:29.161753+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:29.161753+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46108400,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7021,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6843","reported_seq":11438,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:28.588275+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:28.588275+0000","last_peered":"2024-09-11T05:00:28.588275+0000","last_clean":"2024-09-11T05:00:28.588275+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:28.588275+0000","last_undegraded":"2024-09-11T05:00:28.588275+0000","last_fullsized":"2024-09-11T05:00:28.588275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6843,"log_dups_size":0,"ondisk_log_size":6843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45391872,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":653,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7112","reported_seq":7192,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:30.106591+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:30.106591+0000","last_peered":"2024-09-11T05:00:30.106591+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:30.106591+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7112,"log_dups_size":0,"ondisk_log_size":7112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45963856,"num_objects":5611,"num_object_clones":0,"num_object_copies":22444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6969","reported_seq":7026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.169700+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:29.169700+0000","last_peered":"2024-09-11T05:00:29.169700+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:29.169700+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6969,"log_dups_size":0,"ondisk_log_size":6969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6918","reported_seq":7021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.961222+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:29.961222+0000","last_peered":"2024-09-11T05:00:29.961222+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:29.961222+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44979360,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7060","reported_seq":7132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.531269+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:27.531269+0000","last_peered":"2024-09-11T05:00:27.531269+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:27.531269+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830752,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":734824072,"num_objects":89706,"num_object_clones":0,"num_object_copies":358824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89706,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105487,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1369722880,"data_stored":1369722880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111703,"ondisk_log_size":111703,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760500,"kb_used_data":359524,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93611340,"statfs":{"total":96636764160,"available":95858012160,"internally_reserved":0,"allocated":368152576,"data_stored":367972607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,2,5,2,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758964,"kb_used_data":358484,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93612876,"statfs":{"total":96636764160,"available":95859585024,"internally_reserved":0,"allocated":367087616,"data_stored":366907647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532524,"kb_used_data":262108,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93839316,"statfs":{"total":96636764160,"available":96091459584,"internally_reserved":0,"allocated":268398592,"data_stored":268218623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276885202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757700,"kb_used_data":360948,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93614140,"statfs":{"total":96636764160,"available":95860879360,"internally_reserved":0,"allocated":369610752,"data_stored":369430783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,1,0,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368660480,"data_stored":368660480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268103680,"data_stored":268103680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365948928,"data_stored":365948928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367009792,"data_stored":367009792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:32.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:32.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6959", 'reported_seq': 7008, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:29.121209+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:29.121209+0000', 'last_peered': '2024-09-11T05:00:29.121209+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:29.121209+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6959, 'log_dups_size': 0, 'ondisk_log_size': 6959, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45506560, 'num_objects': 5555, 'num_object_clones': 0, 'num_object_copies': 22220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6959, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3244}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:32.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:32.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7038", 'reported_seq': 7115, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:27.347085+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:27.347085+0000', 'last_peered': '2024-09-11T05:00:27.347085+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:27.347085+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7038, 'log_dups_size': 0, 'ondisk_log_size': 7038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46691488, 'num_objects': 5700, 'num_object_clones': 0, 'num_object_copies': 22800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7036, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3426}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:32.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6779", 'reported_seq': 6874, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:30.042336+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:30.042336+0000', 'last_peered': '2024-09-11T05:00:30.042336+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:30.042336+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6779, 'log_dups_size': 0, 'ondisk_log_size': 6779, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44079720, 'num_objects': 5382, 'num_object_clones': 0, 'num_object_copies': 21528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5382, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6778, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3169}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6840", 'reported_seq': 6916, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:27.500072+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:27.500072+0000', 'last_peered': '2024-09-11T05:00:27.500072+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:27.500072+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6840, 'log_dups_size': 0, 'ondisk_log_size': 6840, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45675680, 'num_objects': 5576, 'num_object_clones': 0, 'num_object_copies': 22304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6838, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6916", 'reported_seq': 6985, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:29.106514+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:29.106514+0000', 'last_peered': '2024-09-11T05:00:29.106514+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:29.106514+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6916, 'log_dups_size': 0, 'ondisk_log_size': 6916, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45995168, 'num_objects': 5615, 'num_object_clones': 0, 'num_object_copies': 22460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6914, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3370}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7039", 'reported_seq': 7097, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:29.138403+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:29.138403+0000', 'last_peered': '2024-09-11T05:00:29.138403+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:29.138403+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7039, 'log_dups_size': 0, 'ondisk_log_size': 7039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45883392, 'num_objects': 5601, 'num_object_clones': 0, 'num_object_copies': 22404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7039, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3286}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7005", 'reported_seq': 7077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:27.496590+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:27.496590+0000', 'last_peered': '2024-09-11T05:00:27.496590+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:27.496590+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7005, 'log_dups_size': 0, 'ondisk_log_size': 7005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46896288, 'num_objects': 5725, 'num_object_clones': 0, 'num_object_copies': 22900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7003, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3410}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:32.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7107", 'reported_seq': 8271, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:27.577171+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:27.577171+0000', 'last_peered': '2024-09-11T05:00:27.577171+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:27.577171+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7107, 'log_dups_size': 0, 'ondisk_log_size': 7107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46838944, 'num_objects': 5718, 'num_object_clones': 0, 'num_object_copies': 22872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1813, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7105, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 550, 'num_bytes_recovered': 4464640, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3905}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7116", 'reported_seq': 7193, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:29.115196+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:29.115196+0000', 'last_peered': '2024-09-11T05:00:29.115196+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:29.115196+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7116, 'log_dups_size': 0, 'ondisk_log_size': 7116, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46459200, 'num_objects': 5672, 'num_object_clones': 0, 'num_object_copies': 22688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2389, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7113, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3286}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6978", 'reported_seq': 7072, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:29.959390+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:29.959390+0000', 'last_peered': '2024-09-11T05:00:29.959390+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:29.959390+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6978, 'log_dups_size': 0, 'ondisk_log_size': 6978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45462688, 'num_objects': 5550, 'num_object_clones': 0, 'num_object_copies': 22200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6976, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3238}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7024", 'reported_seq': 7093, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:29.161753+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:29.161753+0000', 'last_peered': '2024-09-11T05:00:29.161753+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:29.161753+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7024, 'log_dups_size': 0, 'ondisk_log_size': 7024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46108400, 'num_objects': 5629, 'num_object_clones': 0, 'num_object_copies': 22516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7021, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3310}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7112", 'reported_seq': 7192, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:30.106591+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:30.106591+0000', 'last_peered': '2024-09-11T05:00:30.106591+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:30.106591+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7112, 'log_dups_size': 0, 'ondisk_log_size': 7112, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45963856, 'num_objects': 5611, 'num_object_clones': 0, 'num_object_copies': 22444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2331, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7111, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:32.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6969", 'reported_seq': 7026, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:29.169700+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:29.169700+0000', 'last_peered': '2024-09-11T05:00:29.169700+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:29.169700+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6969, 'log_dups_size': 0, 'ondisk_log_size': 6969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46060704, 'num_objects': 5623, 'num_object_clones': 0, 'num_object_copies': 22492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2288, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5623, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6967, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:32.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:32.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6918", 'reported_seq': 7021, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:29.961222+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:29.961222+0000', 'last_peered': '2024-09-11T05:00:29.961222+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:29.961222+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6918, 'log_dups_size': 0, 'ondisk_log_size': 6918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44979360, 'num_objects': 5491, 'num_object_clones': 0, 'num_object_copies': 21964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6916, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:32.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:32.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7060", 'reported_seq': 7132, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:27.531269+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:27.531269+0000', 'last_peered': '2024-09-11T05:00:27.531269+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:27.531269+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7060, 'log_dups_size': 0, 'ondisk_log_size': 7060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46830752, 'num_objects': 5717, 'num_object_clones': 0, 'num_object_copies': 22868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7058, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:32.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:32.934 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:32.935 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34034/358824 objects degraded (9.485%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":359,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89706,"data_bytes":734824072,"bytes_used":2877120512,"bytes_avail":383669936128,"bytes_total":386547056640,"degraded_objects":34034,"degraded_total":358824,"degraded_ratio":0.094848728067241886,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":69596,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":835584,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":224},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:32.961 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:32.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:33.570 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:33.570 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:33.599 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-11T05:00:31.720680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":734824072,"num_objects":89706,"num_object_clones":0,"num_object_copies":358824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89706,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105487,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111703,"ondisk_log_size":111703,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809688,"kb_used_data":1341064,"kb_used_omap":14,"kb_used_meta":1468529,"kb_avail":374677672,"statfs":{"total":386547056640,"available":383669936128,"internally_reserved":0,"allocated":1373249536,"data_stored":1372529660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1503774262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,14,3,13,4,0,1,0,0,0,0,2,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22077440,"num_objects":-2695,"num_object_clones":0,"num_object_copies":-10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006085"},"pg_stats":[{"pgid":"1.f","version":"33'6959","reported_seq":7008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.121209+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:29.121209+0000","last_peered":"2024-09-11T05:00:29.121209+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:29.121209+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45506560,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6959,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7038","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.347085+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:27.347085+0000","last_peered":"2024-09-11T05:00:27.347085+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:27.347085+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7038,"log_dups_size":0,"ondisk_log_size":7038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46691488,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7036,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6779","reported_seq":6874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:30.042336+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:30.042336+0000","last_peered":"2024-09-11T05:00:30.042336+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:30.042336+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6779,"log_dups_size":0,"ondisk_log_size":6779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079720,"num_objects":5382,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6778,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6840","reported_seq":6916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.500072+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:27.500072+0000","last_peered":"2024-09-11T05:00:27.500072+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:27.500072+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6840,"log_dups_size":0,"ondisk_log_size":6840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45675680,"num_objects":5576,"num_object_clones":0,"num_object_copies":22304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6916","reported_seq":6985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.106514+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:29.106514+0000","last_peered":"2024-09-11T05:00:29.106514+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:29.106514+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45995168,"num_objects":5615,"num_object_clones":0,"num_object_copies":22460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6914,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7039","reported_seq":7097,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.138403+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:29.138403+0000","last_peered":"2024-09-11T05:00:29.138403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:29.138403+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45883392,"num_objects":5601,"num_object_clones":0,"num_object_copies":22404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7039,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7005","reported_seq":7077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.496590+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:27.496590+0000","last_peered":"2024-09-11T05:00:27.496590+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:27.496590+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46896288,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7003,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3410}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7107","reported_seq":8271,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.577171+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:27.577171+0000","last_peered":"2024-09-11T05:00:27.577171+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:27.577171+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7107,"log_dups_size":0,"ondisk_log_size":7107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46838944,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":550,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7116","reported_seq":7193,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.115196+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:29.115196+0000","last_peered":"2024-09-11T05:00:29.115196+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:29.115196+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46459200,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7113,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6978","reported_seq":7072,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.959390+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:29.959390+0000","last_peered":"2024-09-11T05:00:29.959390+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:29.959390+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45462688,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7024","reported_seq":7093,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.161753+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:29.161753+0000","last_peered":"2024-09-11T05:00:29.161753+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:29.161753+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46108400,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7021,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6843","reported_seq":11438,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:28.588275+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:28.588275+0000","last_peered":"2024-09-11T05:00:28.588275+0000","last_clean":"2024-09-11T05:00:28.588275+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:28.588275+0000","last_undegraded":"2024-09-11T05:00:28.588275+0000","last_fullsized":"2024-09-11T05:00:28.588275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6843,"log_dups_size":0,"ondisk_log_size":6843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45391872,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":653,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7112","reported_seq":7192,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:30.106591+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:30.106591+0000","last_peered":"2024-09-11T05:00:30.106591+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:30.106591+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7112,"log_dups_size":0,"ondisk_log_size":7112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45963856,"num_objects":5611,"num_object_clones":0,"num_object_copies":22444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6969","reported_seq":7026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.169700+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:29.169700+0000","last_peered":"2024-09-11T05:00:29.169700+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:29.169700+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6969,"log_dups_size":0,"ondisk_log_size":6969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6918","reported_seq":7021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:29.961222+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:29.961222+0000","last_peered":"2024-09-11T05:00:29.961222+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:29.961222+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44979360,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7060","reported_seq":7132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:27.531269+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:27.531269+0000","last_peered":"2024-09-11T05:00:27.531269+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:27.531269+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830752,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":734824072,"num_objects":89706,"num_object_clones":0,"num_object_copies":358824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89706,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105487,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":4464640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1369722880,"data_stored":1369722880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111703,"ondisk_log_size":111703,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760500,"kb_used_data":359524,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93611340,"statfs":{"total":96636764160,"available":95858012160,"internally_reserved":0,"allocated":368152576,"data_stored":367972607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,2,5,2,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758964,"kb_used_data":358484,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93612876,"statfs":{"total":96636764160,"available":95859585024,"internally_reserved":0,"allocated":367087616,"data_stored":366907647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532524,"kb_used_data":262108,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93839316,"statfs":{"total":96636764160,"available":96091459584,"internally_reserved":0,"allocated":268398592,"data_stored":268218623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":276885202},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757700,"kb_used_data":360948,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93614140,"statfs":{"total":96636764160,"available":95860879360,"internally_reserved":0,"allocated":369610752,"data_stored":369430783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,1,0,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368660480,"data_stored":368660480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268103680,"data_stored":268103680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365948928,"data_stored":365948928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367009792,"data_stored":367009792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:36.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:37.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:37.255 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:37.273 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-11T05:00:35.722452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":724657800,"num_objects":88465,"num_object_clones":0,"num_object_copies":353860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106728,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":4833280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112944,"ondisk_log_size":112944,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2798728,"kb_used_data":1325932,"kb_used_omap":14,"kb_used_meta":1472689,"kb_avail":374688632,"statfs":{"total":386547056640,"available":383681159168,"internally_reserved":0,"allocated":1357754368,"data_stored":1357034492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1508034102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,9,5,21,5,4,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25853952,"num_objects":-3156,"num_object_clones":0,"num_object_copies":-12624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006135"},"pg_stats":[{"pgid":"1.f","version":"33'7031","reported_seq":7080,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:33.957709+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:33.957709+0000","last_peered":"2024-09-11T05:00:33.957709+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:33.957709+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7031,"log_dups_size":0,"ondisk_log_size":7031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44916736,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7031,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7122","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:32.592466+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:32.592466+0000","last_peered":"2024-09-11T05:00:32.592466+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:32.592466+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7122,"log_dups_size":0,"ondisk_log_size":7122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46003360,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7120,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6865","reported_seq":6960,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.147069+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:35.147069+0000","last_peered":"2024-09-11T05:00:35.147069+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:35.147069+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43375208,"num_objects":5296,"num_object_clones":0,"num_object_copies":21184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5296,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6864,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3083}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6926","reported_seq":7002,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:32.370811+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:32.370811+0000","last_peered":"2024-09-11T05:00:32.370811+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:32.370811+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6926,"log_dups_size":0,"ondisk_log_size":6926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44971168,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6924,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6986","reported_seq":7055,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.187266+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:34.187266+0000","last_peered":"2024-09-11T05:00:34.187266+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:34.187266+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45421728,"num_objects":5545,"num_object_clones":0,"num_object_copies":22180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7116","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.149825+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:34.149825+0000","last_peered":"2024-09-11T05:00:34.149825+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:34.149825+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45252608,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7116,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7072","reported_seq":7144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:32.468526+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:32.468526+0000","last_peered":"2024-09-11T05:00:32.468526+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:32.468526+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7072,"log_dups_size":0,"ondisk_log_size":7072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46347424,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7070,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3343}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7189","reported_seq":8447,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:32.395682+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:32.395682+0000","last_peered":"2024-09-11T05:00:32.395682+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:32.395682+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7189,"log_dups_size":0,"ondisk_log_size":7189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46167200,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1766,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7187,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":4833280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3870}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7192","reported_seq":7268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.200690+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:34.200690+0000","last_peered":"2024-09-11T05:00:34.200690+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:34.200690+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45836608,"num_objects":5596,"num_object_clones":0,"num_object_copies":22384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7049","reported_seq":7143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.963489+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:34.963489+0000","last_peered":"2024-09-11T05:00:34.963489+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:34.963489+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7049,"log_dups_size":0,"ondisk_log_size":7049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44881056,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7047,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3167}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7100","reported_seq":7171,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.217332+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:34.217332+0000","last_peered":"2024-09-11T05:00:34.217332+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:34.217332+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7100,"log_dups_size":0,"ondisk_log_size":7100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45485808,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7097,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6923","reported_seq":11518,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:33.618330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:33.618330+0000","last_peered":"2024-09-11T05:00:33.618330+0000","last_clean":"2024-09-11T05:00:33.618330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:33.618330+0000","last_undegraded":"2024-09-11T05:00:33.618330+0000","last_fullsized":"2024-09-11T05:00:33.618330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44736512,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":733,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7182","reported_seq":7265,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.104898+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:35.104898+0000","last_peered":"2024-09-11T05:00:35.104898+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:35.104898+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7182,"log_dups_size":0,"ondisk_log_size":7182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45390416,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3211}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7039","reported_seq":7096,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.061820+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:34.061820+0000","last_peered":"2024-09-11T05:00:34.061820+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:34.061820+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45487264,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3267}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6998","reported_seq":7100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.130751+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:35.130751+0000","last_peered":"2024-09-11T05:00:35.130751+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:35.130751+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44324000,"num_objects":5411,"num_object_clones":0,"num_object_copies":21644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7154","reported_seq":7226,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:32.609361+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:32.609361+0000","last_peered":"2024-09-11T05:00:32.609361+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:32.609361+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46060704,"num_objects":5623,"num_object_clones":0,"num_object_copies":22492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7152,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":724657800,"num_objects":88465,"num_object_clones":0,"num_object_copies":353860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106728,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":4833280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1354452992,"data_stored":1354452992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112944,"ondisk_log_size":112944,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747236,"kb_used_data":354568,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93624604,"statfs":{"total":96636764160,"available":95871594496,"internally_reserved":0,"allocated":363077632,"data_stored":362897663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,7,0,1,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758136,"kb_used_data":353508,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93613704,"statfs":{"total":96636764160,"available":95860432896,"internally_reserved":0,"allocated":361992192,"data_stored":361812223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":414316997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,8,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536392,"kb_used_data":261816,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93835448,"statfs":{"total":96636764160,"available":96087498752,"internally_reserved":0,"allocated":268099584,"data_stored":267919615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281145042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756964,"kb_used_data":356040,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93614876,"statfs":{"total":96636764160,"available":95861633024,"internally_reserved":0,"allocated":364584960,"data_stored":364404991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363544576,"data_stored":363544576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267804672,"data_stored":267804672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361041920,"data_stored":361041920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362061824,"data_stored":362061824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:37.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:37.901 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:37.918 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-11T05:00:37.723438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":721553032,"num_objects":88086,"num_object_clones":0,"num_object_copies":352344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107107,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113323,"ondisk_log_size":113323,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781380,"kb_used_data":1321060,"kb_used_omap":14,"kb_used_meta":1460209,"kb_avail":374705980,"statfs":{"total":386547056640,"available":383698923520,"internally_reserved":0,"allocated":1352765440,"data_stored":1352045564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1495254582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,4,6,20,6,7,0,0,5,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22806528,"num_objects":-2784,"num_object_clones":0,"num_object_copies":-11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006197"},"pg_stats":[{"pgid":"1.f","version":"33'7031","reported_seq":7080,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:33.957709+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:33.957709+0000","last_peered":"2024-09-11T05:00:33.957709+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:33.957709+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7031,"log_dups_size":0,"ondisk_log_size":7031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44916736,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7031,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7200","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.493827+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:37.493827+0000","last_peered":"2024-09-11T05:00:37.493827+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:37.493827+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45364384,"num_objects":5538,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6865","reported_seq":6960,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.147069+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:35.147069+0000","last_peered":"2024-09-11T05:00:35.147069+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:35.147069+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43375208,"num_objects":5296,"num_object_clones":0,"num_object_copies":21184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5296,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6864,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3083}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7012","reported_seq":7090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.554875+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:37.554875+0000","last_peered":"2024-09-11T05:00:37.554875+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:37.554875+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44266656,"num_objects":5404,"num_object_clones":0,"num_object_copies":21616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6986","reported_seq":7055,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.187266+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:34.187266+0000","last_peered":"2024-09-11T05:00:34.187266+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:34.187266+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45421728,"num_objects":5545,"num_object_clones":0,"num_object_copies":22180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7116","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.149825+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:34.149825+0000","last_peered":"2024-09-11T05:00:34.149825+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:34.149825+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45252608,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7116,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7141","reported_seq":7213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.493434+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:37.493434+0000","last_peered":"2024-09-11T05:00:37.493434+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:37.493434+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7141,"log_dups_size":0,"ondisk_log_size":7141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45782176,"num_objects":5589,"num_object_clones":0,"num_object_copies":22356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3274}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7270","reported_seq":8634,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.567876+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:37.567876+0000","last_peered":"2024-09-11T05:00:37.567876+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:37.567876+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503648,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7192","reported_seq":7268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.200690+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:34.200690+0000","last_peered":"2024-09-11T05:00:34.200690+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:34.200690+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45836608,"num_objects":5596,"num_object_clones":0,"num_object_copies":22384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7049","reported_seq":7143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.963489+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:34.963489+0000","last_peered":"2024-09-11T05:00:34.963489+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:34.963489+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7049,"log_dups_size":0,"ondisk_log_size":7049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44881056,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7047,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3167}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7100","reported_seq":7171,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.217332+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:34.217332+0000","last_peered":"2024-09-11T05:00:34.217332+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:34.217332+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7100,"log_dups_size":0,"ondisk_log_size":7100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45485808,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7097,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6923","reported_seq":11518,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:33.618330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:33.618330+0000","last_peered":"2024-09-11T05:00:33.618330+0000","last_clean":"2024-09-11T05:00:33.618330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:33.618330+0000","last_undegraded":"2024-09-11T05:00:33.618330+0000","last_fullsized":"2024-09-11T05:00:33.618330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44736512,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":733,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7182","reported_seq":7265,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.104898+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:35.104898+0000","last_peered":"2024-09-11T05:00:35.104898+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:35.104898+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7182,"log_dups_size":0,"ondisk_log_size":7182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45390416,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3211}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7039","reported_seq":7096,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.061820+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:34.061820+0000","last_peered":"2024-09-11T05:00:34.061820+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:34.061820+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45487264,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3267}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6998","reported_seq":7100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.130751+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:35.130751+0000","last_peered":"2024-09-11T05:00:35.130751+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:35.130751+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44324000,"num_objects":5411,"num_object_clones":0,"num_object_copies":21644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7219","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.514314+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:37.514314+0000","last_peered":"2024-09-11T05:00:37.514314+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:37.514314+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7219,"log_dups_size":0,"ondisk_log_size":7219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45528224,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7217,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":721553032,"num_objects":88086,"num_object_clones":0,"num_object_copies":352344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107107,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1349574656,"data_stored":1349574656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113323,"ondisk_log_size":113323,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747236,"kb_used_data":354568,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93624604,"statfs":{"total":96636764160,"available":95871594496,"internally_reserved":0,"allocated":363077632,"data_stored":362897663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,7,0,1,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758136,"kb_used_data":353508,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93613704,"statfs":{"total":96636764160,"available":95860432896,"internally_reserved":0,"allocated":361992192,"data_stored":361812223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":414316997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,8,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536392,"kb_used_data":261816,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93835448,"statfs":{"total":96636764160,"available":96087498752,"internally_reserved":0,"allocated":268099584,"data_stored":267919615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281145042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739616,"kb_used_data":351168,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93632224,"statfs":{"total":96636764160,"available":95879397376,"internally_reserved":0,"allocated":359596032,"data_stored":359416063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,4,3,3,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":358666240,"data_stored":358666240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267804672,"data_stored":267804672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361041920,"data_stored":361041920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362061824,"data_stored":362061824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:37.919 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:37.919 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7031", 'reported_seq': 7080, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:33.957709+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:33.957709+0000', 'last_peered': '2024-09-11T05:00:33.957709+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:33.957709+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7031, 'log_dups_size': 0, 'ondisk_log_size': 7031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44916736, 'num_objects': 5483, 'num_object_clones': 0, 'num_object_copies': 21932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7031, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3172}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:37.919 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:37.919 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7200", 'reported_seq': 7280, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.493827+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:37.493827+0000', 'last_peered': '2024-09-11T05:00:37.493827+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:37.493827+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7200, 'log_dups_size': 0, 'ondisk_log_size': 7200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45364384, 'num_objects': 5538, 'num_object_clones': 0, 'num_object_copies': 22152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7198, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6865", 'reported_seq': 6960, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:35.147069+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:35.147069+0000', 'last_peered': '2024-09-11T05:00:35.147069+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:35.147069+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6865, 'log_dups_size': 0, 'ondisk_log_size': 6865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43375208, 'num_objects': 5296, 'num_object_clones': 0, 'num_object_copies': 21184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5296, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6864, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3083}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7012", 'reported_seq': 7090, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.554875+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:37.554875+0000', 'last_peered': '2024-09-11T05:00:37.554875+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:37.554875+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7012, 'log_dups_size': 0, 'ondisk_log_size': 7012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44266656, 'num_objects': 5404, 'num_object_clones': 0, 'num_object_copies': 21616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7010, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6986", 'reported_seq': 7055, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:34.187266+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:34.187266+0000', 'last_peered': '2024-09-11T05:00:34.187266+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:34.187266+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6986, 'log_dups_size': 0, 'ondisk_log_size': 6986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45421728, 'num_objects': 5545, 'num_object_clones': 0, 'num_object_copies': 22180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2247, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6984, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3300}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7116", 'reported_seq': 7174, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:34.149825+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:34.149825+0000', 'last_peered': '2024-09-11T05:00:34.149825+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:34.149825+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7116, 'log_dups_size': 0, 'ondisk_log_size': 7116, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45252608, 'num_objects': 5524, 'num_object_clones': 0, 'num_object_copies': 22096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7116, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3210}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:37.920 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7141", 'reported_seq': 7213, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.493434+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:37.493434+0000', 'last_peered': '2024-09-11T05:00:37.493434+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:37.493434+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7141, 'log_dups_size': 0, 'ondisk_log_size': 7141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45782176, 'num_objects': 5589, 'num_object_clones': 0, 'num_object_copies': 22356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7139, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3274}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7270", 'reported_seq': 8634, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.567876+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:37.567876+0000', 'last_peered': '2024-09-11T05:00:37.567876+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:37.567876+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7270, 'log_dups_size': 0, 'ondisk_log_size': 7270, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45503648, 'num_objects': 5555, 'num_object_clones': 0, 'num_object_copies': 22220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1713, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7268, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 650, 'num_bytes_recovered': 5267456, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3842}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7192", 'reported_seq': 7268, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:34.200690+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:34.200690+0000', 'last_peered': '2024-09-11T05:00:34.200690+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:34.200690+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7192, 'log_dups_size': 0, 'ondisk_log_size': 7192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45836608, 'num_objects': 5596, 'num_object_clones': 0, 'num_object_copies': 22384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2389, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7189, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3210}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7049", 'reported_seq': 7143, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:34.963489+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:34.963489+0000', 'last_peered': '2024-09-11T05:00:34.963489+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:34.963489+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7049, 'log_dups_size': 0, 'ondisk_log_size': 7049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44881056, 'num_objects': 5479, 'num_object_clones': 0, 'num_object_copies': 21916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7047, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3167}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7100", 'reported_seq': 7171, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:34.217332+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:34.217332+0000', 'last_peered': '2024-09-11T05:00:34.217332+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:34.217332+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7100, 'log_dups_size': 0, 'ondisk_log_size': 7100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45485808, 'num_objects': 5553, 'num_object_clones': 0, 'num_object_copies': 22212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7097, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3235}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:37.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:37.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7182", 'reported_seq': 7265, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:35.104898+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:35.104898+0000', 'last_peered': '2024-09-11T05:00:35.104898+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:35.104898+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7182, 'log_dups_size': 0, 'ondisk_log_size': 7182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45390416, 'num_objects': 5541, 'num_object_clones': 0, 'num_object_copies': 22164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2331, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7181, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3211}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:37.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:37.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7039", 'reported_seq': 7096, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:34.061820+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:34.061820+0000', 'last_peered': '2024-09-11T05:00:34.061820+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:34.061820+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7039, 'log_dups_size': 0, 'ondisk_log_size': 7039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45487264, 'num_objects': 5553, 'num_object_clones': 0, 'num_object_copies': 22212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2288, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7037, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3267}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:37.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:37.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6998", 'reported_seq': 7100, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:35.130751+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:35.130751+0000', 'last_peered': '2024-09-11T05:00:35.130751+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:35.130751+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6998, 'log_dups_size': 0, 'ondisk_log_size': 6998, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44324000, 'num_objects': 5411, 'num_object_clones': 0, 'num_object_copies': 21644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5411, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6996, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:37.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:37.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7219", 'reported_seq': 7297, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.514314+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:37.514314+0000', 'last_peered': '2024-09-11T05:00:37.514314+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:37.514314+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7219, 'log_dups_size': 0, 'ondisk_log_size': 7219, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45528224, 'num_objects': 5558, 'num_object_clones': 0, 'num_object_copies': 22232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7217, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:37.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:38.653 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:38.653 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33926/352344 objects degraded (9.629%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":365,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88086,"data_bytes":721553032,"bytes_used":2848133120,"bytes_avail":383698923520,"bytes_total":386547056640,"degraded_objects":33926,"degraded_total":352344,"degraded_ratio":0.096286583566060444,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":103711,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":1245184,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":231},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:38.687 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:38.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:39.372 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:39.373 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:39.393 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-11T05:00:37.723438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":721553032,"num_objects":88086,"num_object_clones":0,"num_object_copies":352344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107107,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113323,"ondisk_log_size":113323,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781380,"kb_used_data":1321060,"kb_used_omap":14,"kb_used_meta":1460209,"kb_avail":374705980,"statfs":{"total":386547056640,"available":383698923520,"internally_reserved":0,"allocated":1352765440,"data_stored":1352045564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1495254582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,4,6,20,6,7,0,0,5,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22806528,"num_objects":-2784,"num_object_clones":0,"num_object_copies":-11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006197"},"pg_stats":[{"pgid":"1.f","version":"33'7031","reported_seq":7080,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:33.957709+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:33.957709+0000","last_peered":"2024-09-11T05:00:33.957709+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:33.957709+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7031,"log_dups_size":0,"ondisk_log_size":7031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44916736,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7031,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7200","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.493827+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:37.493827+0000","last_peered":"2024-09-11T05:00:37.493827+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:37.493827+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45364384,"num_objects":5538,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6865","reported_seq":6960,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.147069+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:35.147069+0000","last_peered":"2024-09-11T05:00:35.147069+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:35.147069+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43375208,"num_objects":5296,"num_object_clones":0,"num_object_copies":21184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5296,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6864,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3083}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7012","reported_seq":7090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.554875+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:37.554875+0000","last_peered":"2024-09-11T05:00:37.554875+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:37.554875+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44266656,"num_objects":5404,"num_object_clones":0,"num_object_copies":21616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6986","reported_seq":7055,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.187266+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:34.187266+0000","last_peered":"2024-09-11T05:00:34.187266+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:34.187266+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45421728,"num_objects":5545,"num_object_clones":0,"num_object_copies":22180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7116","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.149825+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:34.149825+0000","last_peered":"2024-09-11T05:00:34.149825+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:34.149825+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45252608,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7116,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7141","reported_seq":7213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.493434+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:37.493434+0000","last_peered":"2024-09-11T05:00:37.493434+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:37.493434+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7141,"log_dups_size":0,"ondisk_log_size":7141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45782176,"num_objects":5589,"num_object_clones":0,"num_object_copies":22356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3274}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7270","reported_seq":8634,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.567876+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:37.567876+0000","last_peered":"2024-09-11T05:00:37.567876+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:37.567876+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503648,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7192","reported_seq":7268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.200690+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:34.200690+0000","last_peered":"2024-09-11T05:00:34.200690+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:34.200690+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45836608,"num_objects":5596,"num_object_clones":0,"num_object_copies":22384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7049","reported_seq":7143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.963489+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:34.963489+0000","last_peered":"2024-09-11T05:00:34.963489+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:34.963489+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7049,"log_dups_size":0,"ondisk_log_size":7049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44881056,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7047,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3167}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7100","reported_seq":7171,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.217332+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:34.217332+0000","last_peered":"2024-09-11T05:00:34.217332+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:34.217332+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7100,"log_dups_size":0,"ondisk_log_size":7100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45485808,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7097,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6923","reported_seq":11518,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:33.618330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:33.618330+0000","last_peered":"2024-09-11T05:00:33.618330+0000","last_clean":"2024-09-11T05:00:33.618330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:33.618330+0000","last_undegraded":"2024-09-11T05:00:33.618330+0000","last_fullsized":"2024-09-11T05:00:33.618330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44736512,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":733,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7182","reported_seq":7265,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.104898+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:35.104898+0000","last_peered":"2024-09-11T05:00:35.104898+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:35.104898+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7182,"log_dups_size":0,"ondisk_log_size":7182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45390416,"num_objects":5541,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3211}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7039","reported_seq":7096,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:34.061820+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:34.061820+0000","last_peered":"2024-09-11T05:00:34.061820+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:34.061820+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45487264,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3267}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6998","reported_seq":7100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:35.130751+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:35.130751+0000","last_peered":"2024-09-11T05:00:35.130751+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:35.130751+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44324000,"num_objects":5411,"num_object_clones":0,"num_object_copies":21644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7219","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.514314+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:37.514314+0000","last_peered":"2024-09-11T05:00:37.514314+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:37.514314+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7219,"log_dups_size":0,"ondisk_log_size":7219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45528224,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7217,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":721553032,"num_objects":88086,"num_object_clones":0,"num_object_copies":352344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107107,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1349574656,"data_stored":1349574656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113323,"ondisk_log_size":113323,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747236,"kb_used_data":354568,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93624604,"statfs":{"total":96636764160,"available":95871594496,"internally_reserved":0,"allocated":363077632,"data_stored":362897663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,7,0,1,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758136,"kb_used_data":353508,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93613704,"statfs":{"total":96636764160,"available":95860432896,"internally_reserved":0,"allocated":361992192,"data_stored":361812223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":414316997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,8,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536392,"kb_used_data":261816,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93835448,"statfs":{"total":96636764160,"available":96087498752,"internally_reserved":0,"allocated":268099584,"data_stored":267919615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":281145042},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739616,"kb_used_data":351168,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93632224,"statfs":{"total":96636764160,"available":95879397376,"internally_reserved":0,"allocated":359596032,"data_stored":359416063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,4,3,3,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":358666240,"data_stored":358666240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267804672,"data_stored":267804672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361041920,"data_stored":361041920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362061824,"data_stored":362061824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:42.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:43.027 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:43.052 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-11T05:00:41.725435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":714933896,"num_objects":87278,"num_object_clones":0,"num_object_copies":349112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107915,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114131,"ondisk_log_size":114131,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2768872,"kb_used_data":1311304,"kb_used_omap":14,"kb_used_meta":1457457,"kb_avail":374718488,"statfs":{"total":386547056640,"available":383711731712,"internally_reserved":0,"allocated":1342775296,"data_stored":1342055420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1492436534},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,6,9,16,13,9,1,1,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22388736,"num_objects":-2733,"num_object_clones":0,"num_object_copies":-10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2733,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006299"},"pg_stats":[{"pgid":"1.f","version":"33'7117","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.167760+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:39.167760+0000","last_peered":"2024-09-11T05:00:39.167760+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:39.167760+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44212224,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7200","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.493827+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:37.493827+0000","last_peered":"2024-09-11T05:00:37.493827+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:37.493827+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45364384,"num_objects":5538,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6938","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.166339+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:40.166339+0000","last_peered":"2024-09-11T05:00:40.166339+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:40.166339+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42777192,"num_objects":5223,"num_object_clones":0,"num_object_copies":20892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6937,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7012","reported_seq":7090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.554875+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:37.554875+0000","last_peered":"2024-09-11T05:00:37.554875+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:37.554875+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44266656,"num_objects":5404,"num_object_clones":0,"num_object_copies":21616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7048","reported_seq":7123,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.134355+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:39.134355+0000","last_peered":"2024-09-11T05:00:39.134355+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:39.134355+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44913824,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7194","reported_seq":7262,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.171619+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:39.171619+0000","last_peered":"2024-09-11T05:00:39.171619+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:39.171619+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7194,"log_dups_size":0,"ondisk_log_size":7194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44613632,"num_objects":5446,"num_object_clones":0,"num_object_copies":21784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7194,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7141","reported_seq":7213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.493434+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:37.493434+0000","last_peered":"2024-09-11T05:00:37.493434+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:37.493434+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7141,"log_dups_size":0,"ondisk_log_size":7141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45782176,"num_objects":5589,"num_object_clones":0,"num_object_copies":22356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3274}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7270","reported_seq":8634,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.567876+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:37.567876+0000","last_peered":"2024-09-11T05:00:37.567876+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:37.567876+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503648,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7277","reported_seq":7358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:38.896302+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:38.896302+0000","last_peered":"2024-09-11T05:00:38.896302+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:38.896302+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45140288,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7107","reported_seq":7201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.044533+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:40.044533+0000","last_peered":"2024-09-11T05:00:40.044533+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:40.044533+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7107,"log_dups_size":0,"ondisk_log_size":7107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44405920,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7192","reported_seq":7267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.188949+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:39.188949+0000","last_peered":"2024-09-11T05:00:39.188949+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:39.188949+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44732144,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6987","reported_seq":11581,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:38.562701+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:38.562701+0000","last_peered":"2024-09-11T05:00:38.562701+0000","last_clean":"2024-09-11T05:00:38.562701+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:38.562701+0000","last_undegraded":"2024-09-11T05:00:38.562701+0000","last_fullsized":"2024-09-11T05:00:38.562701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44212224,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":797,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7261","reported_seq":7344,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.014414+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:40.014414+0000","last_peered":"2024-09-11T05:00:40.014414+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:40.014414+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7261,"log_dups_size":0,"ondisk_log_size":7261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44743248,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7260,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3132}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7098","reported_seq":7157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:38.827700+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:38.827700+0000","last_peered":"2024-09-11T05:00:38.827700+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:38.827700+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7098,"log_dups_size":0,"ondisk_log_size":7098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45003936,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7096,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3209}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7070","reported_seq":7173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.108125+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:40.108125+0000","last_peered":"2024-09-11T05:00:40.108125+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:40.108125+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7070,"log_dups_size":0,"ondisk_log_size":7070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43734176,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7068,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3031}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7219","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.514314+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:37.514314+0000","last_peered":"2024-09-11T05:00:37.514314+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:37.514314+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7219,"log_dups_size":0,"ondisk_log_size":7219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45528224,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7217,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":714933896,"num_objects":87278,"num_object_clones":0,"num_object_copies":349112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107915,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1339613184,"data_stored":1339613184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114131,"ondisk_log_size":114131,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746572,"kb_used_data":349740,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93625268,"statfs":{"total":96636764160,"available":95872274432,"internally_reserved":0,"allocated":358133760,"data_stored":357953791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,8,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740940,"kb_used_data":348796,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93630900,"statfs":{"total":96636764160,"available":95878041600,"internally_reserved":0,"allocated":357167104,"data_stored":356987135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541744,"kb_used_data":261600,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93830096,"statfs":{"total":96636764160,"available":96082018304,"internally_reserved":0,"allocated":267878400,"data_stored":267698431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286846674},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739616,"kb_used_data":351168,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93632224,"statfs":{"total":96636764160,"available":95879397376,"internally_reserved":0,"allocated":359596032,"data_stored":359416063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,4,3,3,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":358666240,"data_stored":358666240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267595776,"data_stored":267595776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356167680,"data_stored":356167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357183488,"data_stored":357183488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:43.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:43.597 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:43.639 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-11T05:00:41.725435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":714933896,"num_objects":87278,"num_object_clones":0,"num_object_copies":349112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107915,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114131,"ondisk_log_size":114131,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2768872,"kb_used_data":1311304,"kb_used_omap":14,"kb_used_meta":1457457,"kb_avail":374718488,"statfs":{"total":386547056640,"available":383711731712,"internally_reserved":0,"allocated":1342775296,"data_stored":1342055420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1492436534},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,6,9,16,13,9,1,1,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22388736,"num_objects":-2733,"num_object_clones":0,"num_object_copies":-10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2733,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006299"},"pg_stats":[{"pgid":"1.f","version":"33'7117","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.167760+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:39.167760+0000","last_peered":"2024-09-11T05:00:39.167760+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:39.167760+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44212224,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7200","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.493827+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:37.493827+0000","last_peered":"2024-09-11T05:00:37.493827+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:37.493827+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45364384,"num_objects":5538,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6938","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.166339+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:40.166339+0000","last_peered":"2024-09-11T05:00:40.166339+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:40.166339+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42777192,"num_objects":5223,"num_object_clones":0,"num_object_copies":20892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6937,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7012","reported_seq":7090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.554875+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:37.554875+0000","last_peered":"2024-09-11T05:00:37.554875+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:37.554875+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44266656,"num_objects":5404,"num_object_clones":0,"num_object_copies":21616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7048","reported_seq":7123,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.134355+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:39.134355+0000","last_peered":"2024-09-11T05:00:39.134355+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:39.134355+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44913824,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7194","reported_seq":7262,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.171619+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:39.171619+0000","last_peered":"2024-09-11T05:00:39.171619+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:39.171619+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7194,"log_dups_size":0,"ondisk_log_size":7194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44613632,"num_objects":5446,"num_object_clones":0,"num_object_copies":21784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7194,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7141","reported_seq":7213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.493434+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:37.493434+0000","last_peered":"2024-09-11T05:00:37.493434+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:37.493434+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7141,"log_dups_size":0,"ondisk_log_size":7141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45782176,"num_objects":5589,"num_object_clones":0,"num_object_copies":22356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3274}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7270","reported_seq":8634,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.567876+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:37.567876+0000","last_peered":"2024-09-11T05:00:37.567876+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:37.567876+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503648,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7277","reported_seq":7358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:38.896302+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:38.896302+0000","last_peered":"2024-09-11T05:00:38.896302+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:38.896302+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45140288,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7107","reported_seq":7201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.044533+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:40.044533+0000","last_peered":"2024-09-11T05:00:40.044533+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:40.044533+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7107,"log_dups_size":0,"ondisk_log_size":7107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44405920,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7192","reported_seq":7267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.188949+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:39.188949+0000","last_peered":"2024-09-11T05:00:39.188949+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:39.188949+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44732144,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6987","reported_seq":11581,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:38.562701+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:38.562701+0000","last_peered":"2024-09-11T05:00:38.562701+0000","last_clean":"2024-09-11T05:00:38.562701+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:38.562701+0000","last_undegraded":"2024-09-11T05:00:38.562701+0000","last_fullsized":"2024-09-11T05:00:38.562701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44212224,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":797,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7261","reported_seq":7344,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.014414+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:40.014414+0000","last_peered":"2024-09-11T05:00:40.014414+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:40.014414+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7261,"log_dups_size":0,"ondisk_log_size":7261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44743248,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7260,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3132}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7098","reported_seq":7157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:38.827700+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:38.827700+0000","last_peered":"2024-09-11T05:00:38.827700+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:38.827700+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7098,"log_dups_size":0,"ondisk_log_size":7098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45003936,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7096,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3209}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7070","reported_seq":7173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.108125+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:40.108125+0000","last_peered":"2024-09-11T05:00:40.108125+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:40.108125+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7070,"log_dups_size":0,"ondisk_log_size":7070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43734176,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7068,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3031}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7219","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:37.514314+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:37.514314+0000","last_peered":"2024-09-11T05:00:37.514314+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:37.514314+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7219,"log_dups_size":0,"ondisk_log_size":7219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45528224,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7217,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":714933896,"num_objects":87278,"num_object_clones":0,"num_object_copies":349112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107915,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":5267456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1339613184,"data_stored":1339613184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114131,"ondisk_log_size":114131,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746572,"kb_used_data":349740,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93625268,"statfs":{"total":96636764160,"available":95872274432,"internally_reserved":0,"allocated":358133760,"data_stored":357953791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,8,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740940,"kb_used_data":348796,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93630900,"statfs":{"total":96636764160,"available":95878041600,"internally_reserved":0,"allocated":357167104,"data_stored":356987135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541744,"kb_used_data":261600,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93830096,"statfs":{"total":96636764160,"available":96082018304,"internally_reserved":0,"allocated":267878400,"data_stored":267698431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286846674},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739616,"kb_used_data":351168,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93632224,"statfs":{"total":96636764160,"available":95879397376,"internally_reserved":0,"allocated":359596032,"data_stored":359416063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,4,3,3,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":358666240,"data_stored":358666240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267595776,"data_stored":267595776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356167680,"data_stored":356167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357183488,"data_stored":357183488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:43.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:43.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7117", 'reported_seq': 7174, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:39.167760+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:39.167760+0000', 'last_peered': '2024-09-11T05:00:39.167760+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:39.167760+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7117, 'log_dups_size': 0, 'ondisk_log_size': 7117, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44212224, 'num_objects': 5397, 'num_object_clones': 0, 'num_object_copies': 21588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7117, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3090}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7200", 'reported_seq': 7280, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.493827+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:37.493827+0000', 'last_peered': '2024-09-11T05:00:37.493827+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:37.493827+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7200, 'log_dups_size': 0, 'ondisk_log_size': 7200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45364384, 'num_objects': 5538, 'num_object_clones': 0, 'num_object_copies': 22152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7198, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6938", 'reported_seq': 7031, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:40.166339+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:40.166339+0000', 'last_peered': '2024-09-11T05:00:40.166339+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:40.166339+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 6938, 'log_dups_size': 0, 'ondisk_log_size': 6938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42777192, 'num_objects': 5223, 'num_object_clones': 0, 'num_object_copies': 20892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5223, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6937, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3010}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7012", 'reported_seq': 7090, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.554875+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:37.554875+0000', 'last_peered': '2024-09-11T05:00:37.554875+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:37.554875+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7012, 'log_dups_size': 0, 'ondisk_log_size': 7012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44266656, 'num_objects': 5404, 'num_object_clones': 0, 'num_object_copies': 21616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7010, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7048", 'reported_seq': 7123, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:39.134355+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:39.134355+0000', 'last_peered': '2024-09-11T05:00:39.134355+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:39.134355+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7048, 'log_dups_size': 0, 'ondisk_log_size': 7048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44913824, 'num_objects': 5483, 'num_object_clones': 0, 'num_object_copies': 21932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2244, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7046, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3241}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:43.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7194", 'reported_seq': 7262, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:39.171619+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:39.171619+0000', 'last_peered': '2024-09-11T05:00:39.171619+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:39.171619+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7194, 'log_dups_size': 0, 'ondisk_log_size': 7194, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44613632, 'num_objects': 5446, 'num_object_clones': 0, 'num_object_copies': 21784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7194, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3136}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7141", 'reported_seq': 7213, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.493434+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:37.493434+0000', 'last_peered': '2024-09-11T05:00:37.493434+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:37.493434+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7141, 'log_dups_size': 0, 'ondisk_log_size': 7141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45782176, 'num_objects': 5589, 'num_object_clones': 0, 'num_object_copies': 22356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7139, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3274}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7270", 'reported_seq': 8634, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.567876+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:37.567876+0000', 'last_peered': '2024-09-11T05:00:37.567876+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:37.567876+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7270, 'log_dups_size': 0, 'ondisk_log_size': 7270, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45503648, 'num_objects': 5555, 'num_object_clones': 0, 'num_object_copies': 22220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1713, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7268, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 650, 'num_bytes_recovered': 5267456, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3842}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7277", 'reported_seq': 7358, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:38.896302+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:38.896302+0000', 'last_peered': '2024-09-11T05:00:38.896302+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:38.896302+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7277, 'log_dups_size': 0, 'ondisk_log_size': 7277, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45140288, 'num_objects': 5511, 'num_object_clones': 0, 'num_object_copies': 22044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2387, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7274, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3127}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7107", 'reported_seq': 7201, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:40.044533+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:40.044533+0000', 'last_peered': '2024-09-11T05:00:40.044533+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:40.044533+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7107, 'log_dups_size': 0, 'ondisk_log_size': 7107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44405920, 'num_objects': 5421, 'num_object_clones': 0, 'num_object_copies': 21684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7105, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3109}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:43.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7192", 'reported_seq': 7267, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:39.188949+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:39.188949+0000', 'last_peered': '2024-09-11T05:00:39.188949+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:39.188949+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7192, 'log_dups_size': 0, 'ondisk_log_size': 7192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44732144, 'num_objects': 5461, 'num_object_clones': 0, 'num_object_copies': 21844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7189, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3145}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7261", 'reported_seq': 7344, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:40.014414+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:40.014414+0000', 'last_peered': '2024-09-11T05:00:40.014414+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:40.014414+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7261, 'log_dups_size': 0, 'ondisk_log_size': 7261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44743248, 'num_objects': 5462, 'num_object_clones': 0, 'num_object_copies': 21848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2331, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7260, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3132}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7098", 'reported_seq': 7157, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:38.827700+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:38.827700+0000', 'last_peered': '2024-09-11T05:00:38.827700+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:38.827700+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7098, 'log_dups_size': 0, 'ondisk_log_size': 7098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45003936, 'num_objects': 5494, 'num_object_clones': 0, 'num_object_copies': 21976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7096, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7070", 'reported_seq': 7173, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:40.108125+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:40.108125+0000', 'last_peered': '2024-09-11T05:00:40.108125+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:40.108125+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7070, 'log_dups_size': 0, 'ondisk_log_size': 7070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43734176, 'num_objects': 5339, 'num_object_clones': 0, 'num_object_copies': 21356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7068, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3031}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:43.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7219", 'reported_seq': 7297, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:37.514314+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:37.514314+0000', 'last_peered': '2024-09-11T05:00:37.514314+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:37.514314+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7219, 'log_dups_size': 0, 'ondisk_log_size': 7219, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45528224, 'num_objects': 5558, 'num_object_clones': 0, 'num_object_copies': 22232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7217, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:43.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:44.444 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33856/347428 objects degraded (9.745%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":371,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86857,"data_bytes":711485064,"bytes_used":2838949888,"bytes_avail":383708106752,"bytes_total":386547056640,"degraded_objects":33856,"degraded_total":347428,"degraded_ratio":0.097447528696593247,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":103717,"recovering_keys_per_sec":0,"num_objects_recovered":178,"num_bytes_recovered":1245184,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":237},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:44.471 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:44.471 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:45.116 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:45.140 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-11T05:00:43.726203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":711485064,"num_objects":86857,"num_object_clones":0,"num_object_copies":347428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108336,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":815,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114552,"ondisk_log_size":114552,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772412,"kb_used_data":1306540,"kb_used_omap":14,"kb_used_meta":1465777,"kb_avail":374714948,"statfs":{"total":386547056640,"available":383708106752,"internally_reserved":0,"allocated":1337896960,"data_stored":1337177084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1500956214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,6,14,13,13,10,1,1,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23339008,"num_objects":-2849,"num_object_clones":0,"num_object_copies":-11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2849,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005522"},"pg_stats":[{"pgid":"1.f","version":"33'7117","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.167760+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:39.167760+0000","last_peered":"2024-09-11T05:00:39.167760+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:39.167760+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44212224,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7272","reported_seq":7353,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.525584+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:42.525584+0000","last_peered":"2024-09-11T05:00:42.525584+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:42.525584+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774560,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6938","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.166339+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:40.166339+0000","last_peered":"2024-09-11T05:00:40.166339+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:40.166339+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42777192,"num_objects":5223,"num_object_clones":0,"num_object_copies":20892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6937,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7079","reported_seq":7157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.130791+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:42.130791+0000","last_peered":"2024-09-11T05:00:42.130791+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:42.130791+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717792,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7048","reported_seq":7123,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.134355+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:39.134355+0000","last_peered":"2024-09-11T05:00:39.134355+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:39.134355+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44913824,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7194","reported_seq":7262,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.171619+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:39.171619+0000","last_peered":"2024-09-11T05:00:39.171619+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:39.171619+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7194,"log_dups_size":0,"ondisk_log_size":7194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44613632,"num_objects":5446,"num_object_clones":0,"num_object_copies":21784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7194,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7213","reported_seq":7285,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.616337+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:42.616337+0000","last_peered":"2024-09-11T05:00:42.616337+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:42.616337+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45192352,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7211,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3202}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7355","reported_seq":8827,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.544559+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:42.544559+0000","last_peered":"2024-09-11T05:00:42.544559+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:42.544559+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44807328,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7277","reported_seq":7358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:38.896302+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:38.896302+0000","last_peered":"2024-09-11T05:00:38.896302+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:38.896302+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45140288,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7107","reported_seq":7201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.044533+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:40.044533+0000","last_peered":"2024-09-11T05:00:40.044533+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:40.044533+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7107,"log_dups_size":0,"ondisk_log_size":7107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44405920,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7192","reported_seq":7267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:39.188949+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:39.188949+0000","last_peered":"2024-09-11T05:00:39.188949+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:39.188949+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44732144,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7044","reported_seq":11639,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:43.143299+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:43.143299+0000","last_peered":"2024-09-11T05:00:43.143299+0000","last_clean":"2024-09-11T05:00:43.143299+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:43.143299+0000","last_undegraded":"2024-09-11T05:00:43.143299+0000","last_fullsized":"2024-09-11T05:00:43.143299+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43745280,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":854,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7261","reported_seq":7344,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.014414+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:40.014414+0000","last_peered":"2024-09-11T05:00:40.014414+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:40.014414+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7261,"log_dups_size":0,"ondisk_log_size":7261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44743248,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7260,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3132}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7098","reported_seq":7157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:38.827700+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:38.827700+0000","last_peered":"2024-09-11T05:00:38.827700+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:38.827700+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7098,"log_dups_size":0,"ondisk_log_size":7098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45003936,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7096,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3209}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7070","reported_seq":7173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:40.108125+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:40.108125+0000","last_peered":"2024-09-11T05:00:40.108125+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:40.108125+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7070,"log_dups_size":0,"ondisk_log_size":7070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43734176,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7068,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3031}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7287","reported_seq":7363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.602964+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:42.602964+0000","last_peered":"2024-09-11T05:00:42.602964+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:42.602964+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7287,"log_dups_size":0,"ondisk_log_size":7287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44971168,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":711485064,"num_objects":86857,"num_object_clones":0,"num_object_copies":347428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108336,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":815,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1334484992,"data_stored":1334484992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114552,"ondisk_log_size":114552,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746572,"kb_used_data":349740,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93625268,"statfs":{"total":96636764160,"available":95872274432,"internally_reserved":0,"allocated":358133760,"data_stored":357953791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,8,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740940,"kb_used_data":348796,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93630900,"statfs":{"total":96636764160,"available":95878041600,"internally_reserved":0,"allocated":357167104,"data_stored":356987135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545744,"kb_used_data":261440,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93826096,"statfs":{"total":96636764160,"available":96077922304,"internally_reserved":0,"allocated":267714560,"data_stored":267534591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291106514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739156,"kb_used_data":346564,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93632684,"statfs":{"total":96636764160,"available":95879868416,"internally_reserved":0,"allocated":354881536,"data_stored":354701567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,7,1,3,4,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353705984,"data_stored":353705984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267427840,"data_stored":267427840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356167680,"data_stored":356167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357183488,"data_stored":357183488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:48.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:48.786 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:48.786 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:48.817 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-11T05:00:47.727793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705177224,"num_objects":86087,"num_object_clones":0,"num_object_copies":344348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109106,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":826,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115322,"ondisk_log_size":115322,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771344,"kb_used_data":1297156,"kb_used_omap":14,"kb_used_meta":1474097,"kb_avail":374716016,"statfs":{"total":386547056640,"available":383709200384,"internally_reserved":0,"allocated":1328287744,"data_stored":1327567868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1509475894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,20,4,12,6,9,10,1,3,0,0,2,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19480576,"num_objects":-2378,"num_object_clones":0,"num_object_copies":-9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2378,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005340"},"pg_stats":[{"pgid":"1.f","version":"33'7203","reported_seq":7260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.174356+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:44.174356+0000","last_peered":"2024-09-11T05:00:44.174356+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:44.174356+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43507712,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7203,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7272","reported_seq":7353,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.525584+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:42.525584+0000","last_peered":"2024-09-11T05:00:42.525584+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:42.525584+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774560,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7005","reported_seq":7100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.130235+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:45.130235+0000","last_peered":"2024-09-11T05:00:45.130235+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:45.130235+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42228328,"num_objects":5156,"num_object_clones":0,"num_object_copies":20624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7004,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7079","reported_seq":7157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.130791+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:42.130791+0000","last_peered":"2024-09-11T05:00:42.130791+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:42.130791+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717792,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7128","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.239647+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:44.239647+0000","last_peered":"2024-09-11T05:00:44.239647+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:44.239647+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7128,"log_dups_size":0,"ondisk_log_size":7128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44258464,"num_objects":5403,"num_object_clones":0,"num_object_copies":21612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7126,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7271","reported_seq":7338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.109271+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:44.109271+0000","last_peered":"2024-09-11T05:00:44.109271+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:44.109271+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7271,"log_dups_size":0,"ondisk_log_size":7271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43982848,"num_objects":5369,"num_object_clones":0,"num_object_copies":21476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7213","reported_seq":7285,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.616337+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:42.616337+0000","last_peered":"2024-09-11T05:00:42.616337+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:42.616337+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45192352,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7211,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3202}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7355","reported_seq":8827,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.544559+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:42.544559+0000","last_peered":"2024-09-11T05:00:42.544559+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:42.544559+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44807328,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7356","reported_seq":7437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.244162+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:44.244162+0000","last_peered":"2024-09-11T05:00:44.244162+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:44.244162+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44493120,"num_objects":5432,"num_object_clones":0,"num_object_copies":21728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7197","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.209007+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:45.209007+0000","last_peered":"2024-09-11T05:00:45.209007+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:45.209007+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7197,"log_dups_size":0,"ondisk_log_size":7197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7195,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7273","reported_seq":7358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.313702+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:44.313702+0000","last_peered":"2024-09-11T05:00:44.313702+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:44.313702+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44068592,"num_objects":5380,"num_object_clones":0,"num_object_copies":21520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7044","reported_seq":11639,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:43.143299+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:43.143299+0000","last_peered":"2024-09-11T05:00:43.143299+0000","last_clean":"2024-09-11T05:00:43.143299+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:43.143299+0000","last_undegraded":"2024-09-11T05:00:43.143299+0000","last_fullsized":"2024-09-11T05:00:43.143299+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43745280,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":854,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7338","reported_seq":7425,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.159303+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:45.159303+0000","last_peered":"2024-09-11T05:00:45.159303+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:45.159303+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44112464,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3057}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7164","reported_seq":7220,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.303208+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:44.303208+0000","last_peered":"2024-09-11T05:00:44.303208+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:44.303208+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7164,"log_dups_size":0,"ondisk_log_size":7164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44463264,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7162,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3143}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7137","reported_seq":7244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.178927+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:45.178927+0000","last_peered":"2024-09-11T05:00:45.178927+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:45.178927+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7135,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7287","reported_seq":7363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.602964+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:42.602964+0000","last_peered":"2024-09-11T05:00:42.602964+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:42.602964+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7287,"log_dups_size":0,"ondisk_log_size":7287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44971168,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705177224,"num_objects":86087,"num_object_clones":0,"num_object_copies":344348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109106,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":826,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1324863488,"data_stored":1324863488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115322,"ondisk_log_size":115322,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746028,"kb_used_data":345052,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93625812,"statfs":{"total":96636764160,"available":95872831488,"internally_reserved":0,"allocated":353333248,"data_stored":353153279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,2,1,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740416,"kb_used_data":344100,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93631424,"statfs":{"total":96636764160,"available":95878578176,"internally_reserved":0,"allocated":352358400,"data_stored":352178431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,16,0,0,0,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545744,"kb_used_data":261440,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93826096,"statfs":{"total":96636764160,"available":96077922304,"internally_reserved":0,"allocated":267714560,"data_stored":267534591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291106514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739156,"kb_used_data":346564,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93632684,"statfs":{"total":96636764160,"available":95879868416,"internally_reserved":0,"allocated":354881536,"data_stored":354701567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,7,1,3,4,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353705984,"data_stored":353705984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267427840,"data_stored":267427840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351432704,"data_stored":351432704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352296960,"data_stored":352296960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:48.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:49.428 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:49.460 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-11T05:00:47.727793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705177224,"num_objects":86087,"num_object_clones":0,"num_object_copies":344348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109106,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":826,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115322,"ondisk_log_size":115322,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771344,"kb_used_data":1297156,"kb_used_omap":14,"kb_used_meta":1474097,"kb_avail":374716016,"statfs":{"total":386547056640,"available":383709200384,"internally_reserved":0,"allocated":1328287744,"data_stored":1327567868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1509475894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,20,4,12,6,9,10,1,3,0,0,2,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19480576,"num_objects":-2378,"num_object_clones":0,"num_object_copies":-9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2378,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005340"},"pg_stats":[{"pgid":"1.f","version":"33'7203","reported_seq":7260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.174356+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:44.174356+0000","last_peered":"2024-09-11T05:00:44.174356+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:44.174356+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43507712,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7203,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7272","reported_seq":7353,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.525584+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:42.525584+0000","last_peered":"2024-09-11T05:00:42.525584+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:42.525584+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774560,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7005","reported_seq":7100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.130235+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:45.130235+0000","last_peered":"2024-09-11T05:00:45.130235+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:45.130235+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42228328,"num_objects":5156,"num_object_clones":0,"num_object_copies":20624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7004,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7079","reported_seq":7157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.130791+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:42.130791+0000","last_peered":"2024-09-11T05:00:42.130791+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:42.130791+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717792,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7128","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.239647+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:44.239647+0000","last_peered":"2024-09-11T05:00:44.239647+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:44.239647+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7128,"log_dups_size":0,"ondisk_log_size":7128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44258464,"num_objects":5403,"num_object_clones":0,"num_object_copies":21612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7126,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7271","reported_seq":7338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.109271+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:44.109271+0000","last_peered":"2024-09-11T05:00:44.109271+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:44.109271+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7271,"log_dups_size":0,"ondisk_log_size":7271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43982848,"num_objects":5369,"num_object_clones":0,"num_object_copies":21476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7213","reported_seq":7285,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.616337+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:42.616337+0000","last_peered":"2024-09-11T05:00:42.616337+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:42.616337+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45192352,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7211,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3202}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7355","reported_seq":8827,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.544559+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:42.544559+0000","last_peered":"2024-09-11T05:00:42.544559+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:42.544559+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44807328,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7356","reported_seq":7437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.244162+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:44.244162+0000","last_peered":"2024-09-11T05:00:44.244162+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:44.244162+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44493120,"num_objects":5432,"num_object_clones":0,"num_object_copies":21728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7197","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.209007+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:45.209007+0000","last_peered":"2024-09-11T05:00:45.209007+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:45.209007+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7197,"log_dups_size":0,"ondisk_log_size":7197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7195,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7273","reported_seq":7358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.313702+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:44.313702+0000","last_peered":"2024-09-11T05:00:44.313702+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:44.313702+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44068592,"num_objects":5380,"num_object_clones":0,"num_object_copies":21520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7044","reported_seq":11639,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:43.143299+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:43.143299+0000","last_peered":"2024-09-11T05:00:43.143299+0000","last_clean":"2024-09-11T05:00:43.143299+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:43.143299+0000","last_undegraded":"2024-09-11T05:00:43.143299+0000","last_fullsized":"2024-09-11T05:00:43.143299+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43745280,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":854,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7338","reported_seq":7425,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.159303+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:45.159303+0000","last_peered":"2024-09-11T05:00:45.159303+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:45.159303+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44112464,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3057}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7164","reported_seq":7220,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:44.303208+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:44.303208+0000","last_peered":"2024-09-11T05:00:44.303208+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:44.303208+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7164,"log_dups_size":0,"ondisk_log_size":7164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44463264,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7162,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3143}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7137","reported_seq":7244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.178927+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:45.178927+0000","last_peered":"2024-09-11T05:00:45.178927+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:45.178927+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7135,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7287","reported_seq":7363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:42.602964+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:42.602964+0000","last_peered":"2024-09-11T05:00:42.602964+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:42.602964+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7287,"log_dups_size":0,"ondisk_log_size":7287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44971168,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705177224,"num_objects":86087,"num_object_clones":0,"num_object_copies":344348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109106,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":826,"num_bytes_recovered":5709824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1324863488,"data_stored":1324863488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115322,"ondisk_log_size":115322,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746028,"kb_used_data":345052,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93625812,"statfs":{"total":96636764160,"available":95872831488,"internally_reserved":0,"allocated":353333248,"data_stored":353153279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,2,1,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740416,"kb_used_data":344100,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93631424,"statfs":{"total":96636764160,"available":95878578176,"internally_reserved":0,"allocated":352358400,"data_stored":352178431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,16,0,0,0,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545744,"kb_used_data":261440,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93826096,"statfs":{"total":96636764160,"available":96077922304,"internally_reserved":0,"allocated":267714560,"data_stored":267534591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291106514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739156,"kb_used_data":346564,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93632684,"statfs":{"total":96636764160,"available":95879868416,"internally_reserved":0,"allocated":354881536,"data_stored":354701567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,7,1,3,4,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353705984,"data_stored":353705984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267427840,"data_stored":267427840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351432704,"data_stored":351432704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352296960,"data_stored":352296960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:49.461 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:49.461 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7203", 'reported_seq': 7260, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:44.174356+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:44.174356+0000', 'last_peered': '2024-09-11T05:00:44.174356+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:44.174356+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7203, 'log_dups_size': 0, 'ondisk_log_size': 7203, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43507712, 'num_objects': 5311, 'num_object_clones': 0, 'num_object_copies': 21244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7203, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:49.461 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:49.461 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7272", 'reported_seq': 7353, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:42.525584+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:42.525584+0000', 'last_peered': '2024-09-11T05:00:42.525584+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:42.525584+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7272, 'log_dups_size': 0, 'ondisk_log_size': 7272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44774560, 'num_objects': 5466, 'num_object_clones': 0, 'num_object_copies': 21864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7270, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7005", 'reported_seq': 7100, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:45.130235+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:45.130235+0000', 'last_peered': '2024-09-11T05:00:45.130235+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:45.130235+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7005, 'log_dups_size': 0, 'ondisk_log_size': 7005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42228328, 'num_objects': 5156, 'num_object_clones': 0, 'num_object_copies': 20624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5156, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7004, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2943}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7079", 'reported_seq': 7157, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:42.130791+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:42.130791+0000', 'last_peered': '2024-09-11T05:00:42.130791+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:42.130791+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7079, 'log_dups_size': 0, 'ondisk_log_size': 7079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43717792, 'num_objects': 5337, 'num_object_clones': 0, 'num_object_copies': 21348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7077, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3064}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7128", 'reported_seq': 7203, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:44.239647+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:44.239647+0000', 'last_peered': '2024-09-11T05:00:44.239647+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:44.239647+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7128, 'log_dups_size': 0, 'ondisk_log_size': 7128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44258464, 'num_objects': 5403, 'num_object_clones': 0, 'num_object_copies': 21612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2244, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7126, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7271", 'reported_seq': 7338, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:44.109271+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:44.109271+0000', 'last_peered': '2024-09-11T05:00:44.109271+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:44.109271+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7271, 'log_dups_size': 0, 'ondisk_log_size': 7271, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43982848, 'num_objects': 5369, 'num_object_clones': 0, 'num_object_copies': 21476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7271, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:49.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7213", 'reported_seq': 7285, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:42.616337+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:42.616337+0000', 'last_peered': '2024-09-11T05:00:42.616337+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:42.616337+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7213, 'log_dups_size': 0, 'ondisk_log_size': 7213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45192352, 'num_objects': 5517, 'num_object_clones': 0, 'num_object_copies': 22068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7211, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3202}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7355", 'reported_seq': 8827, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:42.544559+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:42.544559+0000', 'last_peered': '2024-09-11T05:00:42.544559+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:42.544559+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7355, 'log_dups_size': 0, 'ondisk_log_size': 7355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44807328, 'num_objects': 5470, 'num_object_clones': 0, 'num_object_copies': 21880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1659, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7353, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 704, 'num_bytes_recovered': 5709824, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3811}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7356", 'reported_seq': 7437, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:44.244162+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:44.244162+0000', 'last_peered': '2024-09-11T05:00:44.244162+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:44.244162+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7356, 'log_dups_size': 0, 'ondisk_log_size': 7356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44493120, 'num_objects': 5432, 'num_object_clones': 0, 'num_object_copies': 21728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2387, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7353, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3048}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7197", 'reported_seq': 7295, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:45.209007+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:45.209007+0000', 'last_peered': '2024-09-11T05:00:45.209007+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:45.209007+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7197, 'log_dups_size': 0, 'ondisk_log_size': 7197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43668640, 'num_objects': 5331, 'num_object_clones': 0, 'num_object_copies': 21324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7195, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3021}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7273", 'reported_seq': 7358, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:44.313702+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:44.313702+0000', 'last_peered': '2024-09-11T05:00:44.313702+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:44.313702+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7273, 'log_dups_size': 0, 'ondisk_log_size': 7273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44068592, 'num_objects': 5380, 'num_object_clones': 0, 'num_object_copies': 21520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7270, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3069}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7338", 'reported_seq': 7425, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:45.159303+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:45.159303+0000', 'last_peered': '2024-09-11T05:00:45.159303+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:45.159303+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7338, 'log_dups_size': 0, 'ondisk_log_size': 7338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44112464, 'num_objects': 5385, 'num_object_clones': 0, 'num_object_copies': 21540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2329, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7337, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7164", 'reported_seq': 7220, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:44.303208+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:44.303208+0000', 'last_peered': '2024-09-11T05:00:44.303208+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:44.303208+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7164, 'log_dups_size': 0, 'ondisk_log_size': 7164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44463264, 'num_objects': 5428, 'num_object_clones': 0, 'num_object_copies': 21712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7162, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3143}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7137", 'reported_seq': 7244, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:45.178927+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:45.178927+0000', 'last_peered': '2024-09-11T05:00:45.178927+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:45.178927+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7137, 'log_dups_size': 0, 'ondisk_log_size': 7137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43185312, 'num_objects': 5272, 'num_object_clones': 0, 'num_object_copies': 21088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7135, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7287", 'reported_seq': 7363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:42.602964+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:42.602964+0000', 'last_peered': '2024-09-11T05:00:42.602964+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:42.602964+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7287, 'log_dups_size': 0, 'ondisk_log_size': 7287, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44971168, 'num_objects': 5490, 'num_object_clones': 0, 'num_object_copies': 21960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7285, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:49.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:50.197 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:50.197 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33777/340772 objects degraded (9.912%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":377,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85193,"data_bytes":697853576,"bytes_used":2827882496,"bytes_avail":383719174144,"bytes_total":386547056640,"degraded_objects":33777,"degraded_total":340772,"degraded_ratio":0.09911905907762375,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":79155,"recovering_keys_per_sec":0,"num_objects_recovered":149,"num_bytes_recovered":950272,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":240},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:50.225 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:50.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:50.858 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:50.860 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:50.905 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-11T05:00:49.728625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":697853576,"num_objects":85193,"num_object_clones":0,"num_object_copies":340772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110000,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":894,"num_bytes_recovered":6217728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116216,"ondisk_log_size":116216,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761604,"kb_used_data":1287404,"kb_used_omap":14,"kb_used_meta":1474097,"kb_avail":374725756,"statfs":{"total":386547056640,"available":383719174144,"internally_reserved":0,"allocated":1318301696,"data_stored":1317581820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1509475894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,19,5,9,10,6,0,0,0,0,3,5],"upper_bound":8192},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23699456,"num_objects":-2893,"num_object_clones":0,"num_object_copies":-11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005187"},"pg_stats":[{"pgid":"1.f","version":"33'7292","reported_seq":7349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.253283+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:49.253283+0000","last_peered":"2024-09-11T05:00:49.253283+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:49.253283+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42778624,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7292,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7365","reported_seq":7448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:47.647048+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:47.647048+0000","last_peered":"2024-09-11T05:00:47.647048+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:47.647048+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7365,"log_dups_size":0,"ondisk_log_size":7365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44012704,"num_objects":5373,"num_object_clones":0,"num_object_copies":21492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7363,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3102}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7005","reported_seq":7100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.130235+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:45.130235+0000","last_peered":"2024-09-11T05:00:45.130235+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:45.130235+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42228328,"num_objects":5156,"num_object_clones":0,"num_object_copies":20624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7004,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7153","reported_seq":7230,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:47.657238+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:47.657238+0000","last_peered":"2024-09-11T05:00:47.657238+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:47.657238+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43111584,"num_objects":5263,"num_object_clones":0,"num_object_copies":21052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2990}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7200","reported_seq":7277,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.329220+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:49.329220+0000","last_peered":"2024-09-11T05:00:49.329220+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:49.329220+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7345","reported_seq":7415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.092220+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:49.092220+0000","last_peered":"2024-09-11T05:00:49.092220+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:49.092220+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43376640,"num_objects":5295,"num_object_clones":0,"num_object_copies":21180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7272","reported_seq":7346,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:47.504738+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:47.504738+0000","last_peered":"2024-09-11T05:00:47.504738+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:47.504738+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44709024,"num_objects":5458,"num_object_clones":0,"num_object_copies":21832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3144}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7421","reported_seq":9017,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:47.616473+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:47.616473+0000","last_peered":"2024-09-11T05:00:47.616473+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:47.616473+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7421,"log_dups_size":0,"ondisk_log_size":7421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44266656,"num_objects":5404,"num_object_clones":0,"num_object_copies":21616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7419,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":766,"num_bytes_recovered":6217728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3807}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7438","reported_seq":7521,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.350719+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:49.350719+0000","last_peered":"2024-09-11T05:00:49.350719+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:49.350719+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7438,"log_dups_size":0,"ondisk_log_size":7438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43821376,"num_objects":5350,"num_object_clones":0,"num_object_copies":21400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7435,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7197","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.209007+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:45.209007+0000","last_peered":"2024-09-11T05:00:45.209007+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:45.209007+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7197,"log_dups_size":0,"ondisk_log_size":7197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7195,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7346","reported_seq":7433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.297998+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:49.297998+0000","last_peered":"2024-09-11T05:00:49.297998+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:49.297998+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7346,"log_dups_size":0,"ondisk_log_size":7346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43470576,"num_objects":5307,"num_object_clones":0,"num_object_copies":21228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7115","reported_seq":11710,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:48.504953+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:48.504953+0000","last_peered":"2024-09-11T05:00:48.504953+0000","last_clean":"2024-09-11T05:00:48.504953+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:48.504953+0000","last_undegraded":"2024-09-11T05:00:48.504953+0000","last_fullsized":"2024-09-11T05:00:48.504953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7115,"log_dups_size":0,"ondisk_log_size":7115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43163648,"num_objects":5269,"num_object_clones":0,"num_object_copies":21076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":925,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7338","reported_seq":7425,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.159303+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:45.159303+0000","last_peered":"2024-09-11T05:00:45.159303+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:45.159303+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44112464,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3057}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7241","reported_seq":7300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.329435+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:49.329435+0000","last_peered":"2024-09-11T05:00:49.329435+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:49.329435+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43832480,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3066}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7137","reported_seq":7244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:45.178927+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:45.178927+0000","last_peered":"2024-09-11T05:00:45.178927+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:45.178927+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7135,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7351","reported_seq":7429,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:47.679540+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:47.679540+0000","last_peered":"2024-09-11T05:00:47.679540+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:47.679540+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44446880,"num_objects":5426,"num_object_clones":0,"num_object_copies":21704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7349,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3120}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":697853576,"num_objects":85193,"num_object_clones":0,"num_object_copies":340772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110000,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":894,"num_bytes_recovered":6217728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314926592,"data_stored":1314926592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116216,"ondisk_log_size":116216,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728768,"kb_used_data":340264,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93643072,"statfs":{"total":96636764160,"available":95890505728,"internally_reserved":0,"allocated":348430336,"data_stored":348250367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,2,6,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740416,"kb_used_data":344100,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93631424,"statfs":{"total":96636764160,"available":95878578176,"internally_reserved":0,"allocated":352358400,"data_stored":352178431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,16,0,0,0,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553748,"kb_used_data":261124,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93818092,"statfs":{"total":96636764160,"available":96069726208,"internally_reserved":0,"allocated":267390976,"data_stored":267211007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299626194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738672,"kb_used_data":341916,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93633168,"statfs":{"total":96636764160,"available":95880364032,"internally_reserved":0,"allocated":350121984,"data_stored":349942015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":348954624,"data_stored":348954624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267112448,"data_stored":267112448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351432704,"data_stored":351432704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347426816,"data_stored":347426816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:53.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:54.567 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:54.600 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-11T05:00:53.730433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692651656,"num_objects":84558,"num_object_clones":0,"num_object_copies":338232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110635,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":953,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116851,"ondisk_log_size":116851,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2760624,"kb_used_data":1278104,"kb_used_omap":14,"kb_used_meta":1482417,"kb_avail":374726736,"statfs":{"total":386547056640,"available":383720177664,"internally_reserved":0,"allocated":1308778496,"data_stored":1308058620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1517995574},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,8,13,9,11,3,0,0,0,1,3,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22282240,"num_objects":-2720,"num_object_clones":0,"num_object_copies":-10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":1425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004998"},"pg_stats":[{"pgid":"1.f","version":"33'7292","reported_seq":7349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.253283+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:49.253283+0000","last_peered":"2024-09-11T05:00:49.253283+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:49.253283+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42778624,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7292,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7430","reported_seq":7513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.654797+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:52.654797+0000","last_peered":"2024-09-11T05:00:52.654797+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:52.654797+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43480224,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7061","reported_seq":7156,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:50.177905+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:50.177905+0000","last_peered":"2024-09-11T05:00:50.177905+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:50.177905+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41769576,"num_objects":5100,"num_object_clones":0,"num_object_copies":20400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7060,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2887}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7222","reported_seq":7300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.683920+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:52.683920+0000","last_peered":"2024-09-11T05:00:52.683920+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:52.683920+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7222,"log_dups_size":0,"ondisk_log_size":7222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546336,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7220,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7200","reported_seq":7277,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.329220+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:49.329220+0000","last_peered":"2024-09-11T05:00:49.329220+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:49.329220+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7345","reported_seq":7415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.092220+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:49.092220+0000","last_peered":"2024-09-11T05:00:49.092220+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:49.092220+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43376640,"num_objects":5295,"num_object_clones":0,"num_object_copies":21180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7342","reported_seq":7416,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.659497+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:52.659497+0000","last_peered":"2024-09-11T05:00:52.659497+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:52.659497+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7342,"log_dups_size":0,"ondisk_log_size":7342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44135584,"num_objects":5388,"num_object_clones":0,"num_object_copies":21552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7340,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7500","reported_seq":9211,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.688658+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:52.688658+0000","last_peered":"2024-09-11T05:00:52.688658+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:52.688658+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7500,"log_dups_size":0,"ondisk_log_size":7500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43619488,"num_objects":5325,"num_object_clones":0,"num_object_copies":21300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":824,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7438","reported_seq":7521,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.350719+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:49.350719+0000","last_peered":"2024-09-11T05:00:49.350719+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:49.350719+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7438,"log_dups_size":0,"ondisk_log_size":7438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43821376,"num_objects":5350,"num_object_clones":0,"num_object_copies":21400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7435,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7269","reported_seq":7367,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:50.213820+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:50.213820+0000","last_peered":"2024-09-11T05:00:50.213820+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:50.213820+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43078816,"num_objects":5259,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7267,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2949}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7346","reported_seq":7433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.297998+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:49.297998+0000","last_peered":"2024-09-11T05:00:49.297998+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:49.297998+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7346,"log_dups_size":0,"ondisk_log_size":7346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43470576,"num_objects":5307,"num_object_clones":0,"num_object_copies":21228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7115","reported_seq":11710,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:48.504953+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:48.504953+0000","last_peered":"2024-09-11T05:00:48.504953+0000","last_clean":"2024-09-11T05:00:48.504953+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:48.504953+0000","last_undegraded":"2024-09-11T05:00:48.504953+0000","last_fullsized":"2024-09-11T05:00:48.504953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7115,"log_dups_size":0,"ondisk_log_size":7115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43163648,"num_objects":5269,"num_object_clones":0,"num_object_copies":21076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":925,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7403","reported_seq":7492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:50.138812+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:50.138812+0000","last_peered":"2024-09-11T05:00:50.138812+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:50.138812+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7403,"log_dups_size":0,"ondisk_log_size":7403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43579984,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7402,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7241","reported_seq":7300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.329435+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:49.329435+0000","last_peered":"2024-09-11T05:00:49.329435+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:49.329435+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43832480,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3066}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7230","reported_seq":7337,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:50.063935+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:50.063935+0000","last_peered":"2024-09-11T05:00:50.063935+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:50.063935+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42423456,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7417","reported_seq":7495,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.546560+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:52.546560+0000","last_peered":"2024-09-11T05:00:52.546560+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:52.546560+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43906208,"num_objects":5360,"num_object_clones":0,"num_object_copies":21440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7415,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":692651656,"num_objects":84558,"num_object_clones":0,"num_object_copies":338232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110635,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":953,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305473024,"data_stored":1305473024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116851,"ondisk_log_size":116851,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728768,"kb_used_data":340264,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93643072,"statfs":{"total":96636764160,"available":95890505728,"internally_reserved":0,"allocated":348430336,"data_stored":348250367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,2,6,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739812,"kb_used_data":339344,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93632028,"statfs":{"total":96636764160,"available":95879196672,"internally_reserved":0,"allocated":347488256,"data_stored":347308287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,4,4,1,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553748,"kb_used_data":261124,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93818092,"statfs":{"total":96636764160,"available":96069726208,"internally_reserved":0,"allocated":267390976,"data_stored":267211007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299626194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738296,"kb_used_data":337372,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93633544,"statfs":{"total":96636764160,"available":95880749056,"internally_reserved":0,"allocated":345468928,"data_stored":345288959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,2,4,2,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344260608,"data_stored":344260608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267112448,"data_stored":267112448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346673152,"data_stored":346673152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347426816,"data_stored":347426816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:54.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:55.235 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:55.235 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:55.263 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-11T05:00:53.730433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692651656,"num_objects":84558,"num_object_clones":0,"num_object_copies":338232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110635,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":953,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116851,"ondisk_log_size":116851,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2760624,"kb_used_data":1278104,"kb_used_omap":14,"kb_used_meta":1482417,"kb_avail":374726736,"statfs":{"total":386547056640,"available":383720177664,"internally_reserved":0,"allocated":1308778496,"data_stored":1308058620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1517995574},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,8,13,9,11,3,0,0,0,1,3,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22282240,"num_objects":-2720,"num_object_clones":0,"num_object_copies":-10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":1425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004998"},"pg_stats":[{"pgid":"1.f","version":"33'7292","reported_seq":7349,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.253283+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:49.253283+0000","last_peered":"2024-09-11T05:00:49.253283+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:49.253283+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42778624,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7292,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7430","reported_seq":7513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.654797+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:52.654797+0000","last_peered":"2024-09-11T05:00:52.654797+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:52.654797+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43480224,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7061","reported_seq":7156,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:50.177905+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:50.177905+0000","last_peered":"2024-09-11T05:00:50.177905+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:50.177905+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41769576,"num_objects":5100,"num_object_clones":0,"num_object_copies":20400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7060,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2887}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7222","reported_seq":7300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.683920+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:52.683920+0000","last_peered":"2024-09-11T05:00:52.683920+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:52.683920+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7222,"log_dups_size":0,"ondisk_log_size":7222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546336,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7220,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7200","reported_seq":7277,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.329220+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:49.329220+0000","last_peered":"2024-09-11T05:00:49.329220+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:49.329220+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43668640,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7345","reported_seq":7415,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.092220+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:49.092220+0000","last_peered":"2024-09-11T05:00:49.092220+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:49.092220+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43376640,"num_objects":5295,"num_object_clones":0,"num_object_copies":21180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7342","reported_seq":7416,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.659497+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:52.659497+0000","last_peered":"2024-09-11T05:00:52.659497+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:52.659497+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7342,"log_dups_size":0,"ondisk_log_size":7342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44135584,"num_objects":5388,"num_object_clones":0,"num_object_copies":21552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7340,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7500","reported_seq":9211,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.688658+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:52.688658+0000","last_peered":"2024-09-11T05:00:52.688658+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:52.688658+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7500,"log_dups_size":0,"ondisk_log_size":7500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43619488,"num_objects":5325,"num_object_clones":0,"num_object_copies":21300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":824,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7438","reported_seq":7521,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.350719+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:49.350719+0000","last_peered":"2024-09-11T05:00:49.350719+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:49.350719+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7438,"log_dups_size":0,"ondisk_log_size":7438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43821376,"num_objects":5350,"num_object_clones":0,"num_object_copies":21400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7435,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7269","reported_seq":7367,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:50.213820+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:50.213820+0000","last_peered":"2024-09-11T05:00:50.213820+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:50.213820+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43078816,"num_objects":5259,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7267,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2949}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7346","reported_seq":7433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.297998+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:49.297998+0000","last_peered":"2024-09-11T05:00:49.297998+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:49.297998+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7346,"log_dups_size":0,"ondisk_log_size":7346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43470576,"num_objects":5307,"num_object_clones":0,"num_object_copies":21228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7115","reported_seq":11710,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:48.504953+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:48.504953+0000","last_peered":"2024-09-11T05:00:48.504953+0000","last_clean":"2024-09-11T05:00:48.504953+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:48.504953+0000","last_undegraded":"2024-09-11T05:00:48.504953+0000","last_fullsized":"2024-09-11T05:00:48.504953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7115,"log_dups_size":0,"ondisk_log_size":7115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43163648,"num_objects":5269,"num_object_clones":0,"num_object_copies":21076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":925,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7403","reported_seq":7492,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:50.138812+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:50.138812+0000","last_peered":"2024-09-11T05:00:50.138812+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:50.138812+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7403,"log_dups_size":0,"ondisk_log_size":7403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43579984,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7402,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7241","reported_seq":7300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:49.329435+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:49.329435+0000","last_peered":"2024-09-11T05:00:49.329435+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:49.329435+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43832480,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3066}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7230","reported_seq":7337,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:50.063935+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:50.063935+0000","last_peered":"2024-09-11T05:00:50.063935+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:50.063935+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42423456,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7417","reported_seq":7495,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.546560+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:52.546560+0000","last_peered":"2024-09-11T05:00:52.546560+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:52.546560+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43906208,"num_objects":5360,"num_object_clones":0,"num_object_copies":21440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7415,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":692651656,"num_objects":84558,"num_object_clones":0,"num_object_copies":338232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110635,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":953,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305473024,"data_stored":1305473024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116851,"ondisk_log_size":116851,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728768,"kb_used_data":340264,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93643072,"statfs":{"total":96636764160,"available":95890505728,"internally_reserved":0,"allocated":348430336,"data_stored":348250367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,2,6,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739812,"kb_used_data":339344,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93632028,"statfs":{"total":96636764160,"available":95879196672,"internally_reserved":0,"allocated":347488256,"data_stored":347308287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,4,4,1,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553748,"kb_used_data":261124,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93818092,"statfs":{"total":96636764160,"available":96069726208,"internally_reserved":0,"allocated":267390976,"data_stored":267211007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299626194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738296,"kb_used_data":337372,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93633544,"statfs":{"total":96636764160,"available":95880749056,"internally_reserved":0,"allocated":345468928,"data_stored":345288959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,2,4,2,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344260608,"data_stored":344260608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267112448,"data_stored":267112448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346673152,"data_stored":346673152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347426816,"data_stored":347426816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:55.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7292", 'reported_seq': 7349, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:49.253283+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:49.253283+0000', 'last_peered': '2024-09-11T05:00:49.253283+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:49.253283+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7292, 'log_dups_size': 0, 'ondisk_log_size': 7292, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42778624, 'num_objects': 5222, 'num_object_clones': 0, 'num_object_copies': 20888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7292, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2915}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7430", 'reported_seq': 7513, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:52.654797+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:52.654797+0000', 'last_peered': '2024-09-11T05:00:52.654797+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:52.654797+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7430, 'log_dups_size': 0, 'ondisk_log_size': 7430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43480224, 'num_objects': 5308, 'num_object_clones': 0, 'num_object_copies': 21232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7428, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7061", 'reported_seq': 7156, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:50.177905+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:50.177905+0000', 'last_peered': '2024-09-11T05:00:50.177905+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:50.177905+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7061, 'log_dups_size': 0, 'ondisk_log_size': 7061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41769576, 'num_objects': 5100, 'num_object_clones': 0, 'num_object_copies': 20400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5100, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7060, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2887}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7222", 'reported_seq': 7300, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:52.683920+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:52.683920+0000', 'last_peered': '2024-09-11T05:00:52.683920+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:52.683920+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7222, 'log_dups_size': 0, 'ondisk_log_size': 7222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42546336, 'num_objects': 5194, 'num_object_clones': 0, 'num_object_copies': 20776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7220, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2921}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:00:55.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7200", 'reported_seq': 7277, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:49.329220+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:49.329220+0000', 'last_peered': '2024-09-11T05:00:49.329220+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:49.329220+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7200, 'log_dups_size': 0, 'ondisk_log_size': 7200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43668640, 'num_objects': 5331, 'num_object_clones': 0, 'num_object_copies': 21324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2243, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7198, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3090}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7345", 'reported_seq': 7415, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:49.092220+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:49.092220+0000', 'last_peered': '2024-09-11T05:00:49.092220+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:49.092220+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7345, 'log_dups_size': 0, 'ondisk_log_size': 7345, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43376640, 'num_objects': 5295, 'num_object_clones': 0, 'num_object_copies': 21180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7345, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2986}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7342", 'reported_seq': 7416, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:52.659497+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:52.659497+0000', 'last_peered': '2024-09-11T05:00:52.659497+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:52.659497+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7342, 'log_dups_size': 0, 'ondisk_log_size': 7342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44135584, 'num_objects': 5388, 'num_object_clones': 0, 'num_object_copies': 21552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7340, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3074}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7500", 'reported_seq': 9211, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:52.688658+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:52.688658+0000', 'last_peered': '2024-09-11T05:00:52.688658+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:52.688658+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7500, 'log_dups_size': 0, 'ondisk_log_size': 7500, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43619488, 'num_objects': 5325, 'num_object_clones': 0, 'num_object_copies': 21300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1539, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5325, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7498, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 824, 'num_bytes_recovered': 6692864, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7438", 'reported_seq': 7521, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:49.350719+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:49.350719+0000', 'last_peered': '2024-09-11T05:00:49.350719+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:49.350719+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7438, 'log_dups_size': 0, 'ondisk_log_size': 7438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43821376, 'num_objects': 5350, 'num_object_clones': 0, 'num_object_copies': 21400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7435, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2967}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:00:55.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7269", 'reported_seq': 7367, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:50.213820+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:50.213820+0000', 'last_peered': '2024-09-11T05:00:50.213820+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:50.213820+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7269, 'log_dups_size': 0, 'ondisk_log_size': 7269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43078816, 'num_objects': 5259, 'num_object_clones': 0, 'num_object_copies': 21036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7267, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2949}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7346", 'reported_seq': 7433, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:49.297998+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:49.297998+0000', 'last_peered': '2024-09-11T05:00:49.297998+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:49.297998+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7346, 'log_dups_size': 0, 'ondisk_log_size': 7346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43470576, 'num_objects': 5307, 'num_object_clones': 0, 'num_object_copies': 21228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7343, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2997}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7403", 'reported_seq': 7492, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:50.138812+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:50.138812+0000', 'last_peered': '2024-09-11T05:00:50.138812+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:50.138812+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7403, 'log_dups_size': 0, 'ondisk_log_size': 7403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43579984, 'num_objects': 5320, 'num_object_clones': 0, 'num_object_copies': 21280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2328, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7402, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7241", 'reported_seq': 7300, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:49.329435+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:49.329435+0000', 'last_peered': '2024-09-11T05:00:49.329435+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:49.329435+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7241, 'log_dups_size': 0, 'ondisk_log_size': 7241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43832480, 'num_objects': 5351, 'num_object_clones': 0, 'num_object_copies': 21404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7239, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3066}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7230", 'reported_seq': 7337, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:50.063935+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:50.063935+0000', 'last_peered': '2024-09-11T05:00:50.063935+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:50.063935+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7230, 'log_dups_size': 0, 'ondisk_log_size': 7230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42423456, 'num_objects': 5179, 'num_object_clones': 0, 'num_object_copies': 20716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7228, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2873}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:00:55.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:00:55.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7417", 'reported_seq': 7495, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:52.546560+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:52.546560+0000', 'last_peered': '2024-09-11T05:00:52.546560+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:52.546560+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7417, 'log_dups_size': 0, 'ondisk_log_size': 7417, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43906208, 'num_objects': 5360, 'num_object_clones': 0, 'num_object_copies': 21440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7415, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:00:55.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:00:55.998 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:55.998 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33718/338232 objects degraded (9.969%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":382,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84558,"data_bytes":692651656,"bytes_used":2826878976,"bytes_avail":383720177664,"bytes_total":386547056640,"degraded_objects":33718,"degraded_total":338232,"degraded_ratio":0.099688970883890352,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":118734,"recovering_keys_per_sec":0,"num_objects_recovered":192,"num_bytes_recovered":1425408,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":226},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:00:56.021 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:00:56.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:00:56.690 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:00:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:00:56.710 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-11T05:00:55.731264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":686065288,"num_objects":83754,"num_object_clones":0,"num_object_copies":335016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111439,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117655,"ondisk_log_size":117655,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766304,"kb_used_data":1268500,"kb_used_omap":14,"kb_used_meta":1497713,"kb_avail":374721056,"statfs":{"total":386547056640,"available":383714361344,"internally_reserved":0,"allocated":1298944000,"data_stored":1298224124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1533658678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,8,8,11,10,5,0,0,0,0,2,1,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25419776,"num_objects":-3103,"num_object_clones":0,"num_object_copies":-12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005061"},"pg_stats":[{"pgid":"1.f","version":"33'7356","reported_seq":7413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.329002+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:54.329002+0000","last_peered":"2024-09-11T05:00:54.329002+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:54.329002+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42254336,"num_objects":5158,"num_object_clones":0,"num_object_copies":20632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7430","reported_seq":7513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.654797+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:52.654797+0000","last_peered":"2024-09-11T05:00:52.654797+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:52.654797+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43480224,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7124","reported_seq":7219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.964839+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:54.964839+0000","last_peered":"2024-09-11T05:00:54.964839+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:54.964839+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41253480,"num_objects":5037,"num_object_clones":0,"num_object_copies":20148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7123,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7222","reported_seq":7300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.683920+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:52.683920+0000","last_peered":"2024-09-11T05:00:52.683920+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:52.683920+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7222,"log_dups_size":0,"ondisk_log_size":7222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546336,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7220,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7280","reported_seq":7357,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.141167+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:54.141167+0000","last_peered":"2024-09-11T05:00:54.141167+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:54.141167+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43013280,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7413","reported_seq":7483,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.288755+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:54.288755+0000","last_peered":"2024-09-11T05:00:54.288755+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:54.288755+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42819584,"num_objects":5227,"num_object_clones":0,"num_object_copies":20908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7413,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7342","reported_seq":7416,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.659497+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:52.659497+0000","last_peered":"2024-09-11T05:00:52.659497+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:52.659497+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7342,"log_dups_size":0,"ondisk_log_size":7342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44135584,"num_objects":5388,"num_object_clones":0,"num_object_copies":21552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7340,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7500","reported_seq":9211,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.688658+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:52.688658+0000","last_peered":"2024-09-11T05:00:52.688658+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:52.688658+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7500,"log_dups_size":0,"ondisk_log_size":7500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43619488,"num_objects":5325,"num_object_clones":0,"num_object_copies":21300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":824,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7515","reported_seq":7600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.321754+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:54.321754+0000","last_peered":"2024-09-11T05:00:54.321754+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:54.321754+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43190592,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7512,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7338","reported_seq":7436,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.084116+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:55.084116+0000","last_peered":"2024-09-11T05:00:55.084116+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:55.084116+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513568,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7336,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7432","reported_seq":7521,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.353098+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:54.353098+0000","last_peered":"2024-09-11T05:00:54.353098+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:54.353098+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7432,"log_dups_size":0,"ondisk_log_size":7432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42766064,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7429,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2912}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7188","reported_seq":11783,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:53.609026+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:53.609026+0000","last_peered":"2024-09-11T05:00:53.609026+0000","last_clean":"2024-09-11T05:00:53.609026+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:53.609026+0000","last_undegraded":"2024-09-11T05:00:53.609026+0000","last_fullsized":"2024-09-11T05:00:53.609026+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42565632,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":998,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7461","reported_seq":7550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.062908+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:55.062908+0000","last_peered":"2024-09-11T05:00:55.062908+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:55.062908+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7461,"log_dups_size":0,"ondisk_log_size":7461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43104848,"num_objects":5262,"num_object_clones":0,"num_object_copies":21048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7331","reported_seq":7393,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.339520+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:54.339520+0000","last_peered":"2024-09-11T05:00:54.339520+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:54.339520+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7331,"log_dups_size":0,"ondisk_log_size":7331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095200,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7329,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7306","reported_seq":7413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.236847+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:55.236847+0000","last_peered":"2024-09-11T05:00:55.236847+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:55.236847+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41800864,"num_objects":5103,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7417","reported_seq":7495,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:52.546560+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:52.546560+0000","last_peered":"2024-09-11T05:00:52.546560+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:52.546560+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43906208,"num_objects":5360,"num_object_clones":0,"num_object_copies":21440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7415,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":686065288,"num_objects":83754,"num_object_clones":0,"num_object_copies":335016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111439,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1295556608,"data_stored":1295556608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117655,"ondisk_log_size":117655,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746368,"kb_used_data":335664,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93625472,"statfs":{"total":96636764160,"available":95872483328,"internally_reserved":0,"allocated":343719936,"data_stored":343539967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":420540112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,3,3,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740624,"kb_used_data":334592,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93631216,"statfs":{"total":96636764160,"available":95878365184,"internally_reserved":0,"allocated":342622208,"data_stored":342442239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,1,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541016,"kb_used_data":260872,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93830824,"statfs":{"total":96636764160,"available":96082763776,"internally_reserved":0,"allocated":267132928,"data_stored":266952959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286846674},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738296,"kb_used_data":337372,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93633544,"statfs":{"total":96636764160,"available":95880749056,"internally_reserved":0,"allocated":345468928,"data_stored":345288959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,2,4,2,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344260608,"data_stored":344260608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266846208,"data_stored":266846208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341856256,"data_stored":341856256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":342593536,"data_stored":342593536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:00:59.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:00.401 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:00.425 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-11T05:00:59.732667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":678774408,"num_objects":82864,"num_object_clones":0,"num_object_copies":331456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82864,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112329,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118545,"ondisk_log_size":118545,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770756,"kb_used_data":1259076,"kb_used_omap":14,"kb_used_meta":1511601,"kb_avail":374716604,"statfs":{"total":386547056640,"available":383709802496,"internally_reserved":0,"allocated":1289293824,"data_stored":1288573948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1547879990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,11,6,8,3,0,0,0,0,0,1,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26402816,"num_objects":-3223,"num_object_clones":0,"num_object_copies":-12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004874"},"pg_stats":[{"pgid":"1.f","version":"33'7412","reported_seq":7471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.191514+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:59.191514+0000","last_peered":"2024-09-11T05:00:59.191514+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:59.191514+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41795584,"num_objects":5102,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7507","reported_seq":7590,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.594192+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:57.594192+0000","last_peered":"2024-09-11T05:00:57.594192+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:57.594192+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7507,"log_dups_size":0,"ondisk_log_size":7507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42849440,"num_objects":5231,"num_object_clones":0,"num_object_copies":20924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7124","reported_seq":7219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.964839+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:54.964839+0000","last_peered":"2024-09-11T05:00:54.964839+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:54.964839+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41253480,"num_objects":5037,"num_object_clones":0,"num_object_copies":20148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7123,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7295","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.660705+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:57.660705+0000","last_peered":"2024-09-11T05:00:57.660705+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:57.660705+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7295,"log_dups_size":0,"ondisk_log_size":7295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948320,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7293,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7353","reported_seq":7430,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.398898+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:59.398898+0000","last_peered":"2024-09-11T05:00:59.398898+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:59.398898+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7353,"log_dups_size":0,"ondisk_log_size":7353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42415264,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7490","reported_seq":7562,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.273244+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:59.273244+0000","last_peered":"2024-09-11T05:00:59.273244+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:59.273244+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42188800,"num_objects":5150,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7427","reported_seq":7501,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.666869+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:57.666869+0000","last_peered":"2024-09-11T05:00:57.666869+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:57.666869+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439264,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7425,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2989}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7564","reported_seq":9376,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.739387+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:57.739387+0000","last_peered":"2024-09-11T05:00:57.739387+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:57.739387+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7564,"log_dups_size":0,"ondisk_log_size":7564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095200,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7606","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.363902+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:59.363902+0000","last_peered":"2024-09-11T05:00:59.363902+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:59.363902+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42445120,"num_objects":5182,"num_object_clones":0,"num_object_copies":20728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7603,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7338","reported_seq":7436,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.084116+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:55.084116+0000","last_peered":"2024-09-11T05:00:55.084116+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:55.084116+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513568,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7336,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7499","reported_seq":7588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.368736+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:59.368736+0000","last_peered":"2024-09-11T05:00:59.368736+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:59.368736+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42217200,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7249","reported_seq":11844,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:58.704754+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:58.704754+0000","last_peered":"2024-09-11T05:00:58.704754+0000","last_clean":"2024-09-11T05:00:58.704754+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:58.704754+0000","last_undegraded":"2024-09-11T05:00:58.704754+0000","last_fullsized":"2024-09-11T05:00:58.704754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42065920,"num_objects":5135,"num_object_clones":0,"num_object_copies":20540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1059,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7461","reported_seq":7550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.062908+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:55.062908+0000","last_peered":"2024-09-11T05:00:55.062908+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:55.062908+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7461,"log_dups_size":0,"ondisk_log_size":7461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43104848,"num_objects":5262,"num_object_clones":0,"num_object_copies":21048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7409","reported_seq":7486,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.162794+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:59.162794+0000","last_peered":"2024-09-11T05:00:59.162794+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:59.162794+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7409,"log_dups_size":0,"ondisk_log_size":7409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42456224,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7306","reported_seq":7413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.236847+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:55.236847+0000","last_peered":"2024-09-11T05:00:55.236847+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:55.236847+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41800864,"num_objects":5103,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7505","reported_seq":7581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.765992+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:57.765992+0000","last_peered":"2024-09-11T05:00:57.765992+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:57.765992+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":678774408,"num_objects":82864,"num_object_clones":0,"num_object_copies":331456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82864,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112329,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1285799936,"data_stored":1285799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118545,"ondisk_log_size":118545,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745892,"kb_used_data":331028,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93625948,"statfs":{"total":96636764160,"available":95872970752,"internally_reserved":0,"allocated":338972672,"data_stored":338792703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":424799952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740624,"kb_used_data":334592,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93631216,"statfs":{"total":96636764160,"available":95878365184,"internally_reserved":0,"allocated":342622208,"data_stored":342442239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,1,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544976,"kb_used_data":260672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93826864,"statfs":{"total":96636764160,"available":96078708736,"internally_reserved":0,"allocated":266928128,"data_stored":266748159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291106514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739264,"kb_used_data":332784,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93632576,"statfs":{"total":96636764160,"available":95879757824,"internally_reserved":0,"allocated":340770816,"data_stored":340590847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,4,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339480576,"data_stored":339480576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266637312,"data_stored":266637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341856256,"data_stored":341856256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":337825792,"data_stored":337825792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:00.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:01.096 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-11T05:00:59.732667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":678774408,"num_objects":82864,"num_object_clones":0,"num_object_copies":331456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82864,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112329,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118545,"ondisk_log_size":118545,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770756,"kb_used_data":1259076,"kb_used_omap":14,"kb_used_meta":1511601,"kb_avail":374716604,"statfs":{"total":386547056640,"available":383709802496,"internally_reserved":0,"allocated":1289293824,"data_stored":1288573948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1547879990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,11,6,8,3,0,0,0,0,0,1,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26402816,"num_objects":-3223,"num_object_clones":0,"num_object_copies":-12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004874"},"pg_stats":[{"pgid":"1.f","version":"33'7412","reported_seq":7471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.191514+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:59.191514+0000","last_peered":"2024-09-11T05:00:59.191514+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:59.191514+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41795584,"num_objects":5102,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7507","reported_seq":7590,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.594192+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:57.594192+0000","last_peered":"2024-09-11T05:00:57.594192+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:57.594192+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7507,"log_dups_size":0,"ondisk_log_size":7507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42849440,"num_objects":5231,"num_object_clones":0,"num_object_copies":20924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7124","reported_seq":7219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:54.964839+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:00:54.964839+0000","last_peered":"2024-09-11T05:00:54.964839+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:00:54.964839+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41253480,"num_objects":5037,"num_object_clones":0,"num_object_copies":20148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7123,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7295","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.660705+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:57.660705+0000","last_peered":"2024-09-11T05:00:57.660705+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:57.660705+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7295,"log_dups_size":0,"ondisk_log_size":7295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948320,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7293,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7353","reported_seq":7430,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.398898+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:59.398898+0000","last_peered":"2024-09-11T05:00:59.398898+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:59.398898+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7353,"log_dups_size":0,"ondisk_log_size":7353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42415264,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7490","reported_seq":7562,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.273244+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:59.273244+0000","last_peered":"2024-09-11T05:00:59.273244+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:59.273244+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42188800,"num_objects":5150,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7427","reported_seq":7501,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.666869+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:57.666869+0000","last_peered":"2024-09-11T05:00:57.666869+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:57.666869+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439264,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7425,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2989}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7564","reported_seq":9376,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.739387+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:57.739387+0000","last_peered":"2024-09-11T05:00:57.739387+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:57.739387+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7564,"log_dups_size":0,"ondisk_log_size":7564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095200,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7606","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.363902+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:59.363902+0000","last_peered":"2024-09-11T05:00:59.363902+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:59.363902+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42445120,"num_objects":5182,"num_object_clones":0,"num_object_copies":20728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7603,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7338","reported_seq":7436,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.084116+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:00:55.084116+0000","last_peered":"2024-09-11T05:00:55.084116+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:00:55.084116+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513568,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7336,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7499","reported_seq":7588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.368736+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:59.368736+0000","last_peered":"2024-09-11T05:00:59.368736+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:59.368736+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42217200,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7249","reported_seq":11844,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:58.704754+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:58.704754+0000","last_peered":"2024-09-11T05:00:58.704754+0000","last_clean":"2024-09-11T05:00:58.704754+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:58.704754+0000","last_undegraded":"2024-09-11T05:00:58.704754+0000","last_fullsized":"2024-09-11T05:00:58.704754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42065920,"num_objects":5135,"num_object_clones":0,"num_object_copies":20540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1059,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7461","reported_seq":7550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.062908+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:00:55.062908+0000","last_peered":"2024-09-11T05:00:55.062908+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:00:55.062908+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7461,"log_dups_size":0,"ondisk_log_size":7461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43104848,"num_objects":5262,"num_object_clones":0,"num_object_copies":21048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2935}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7409","reported_seq":7486,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.162794+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:59.162794+0000","last_peered":"2024-09-11T05:00:59.162794+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:59.162794+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7409,"log_dups_size":0,"ondisk_log_size":7409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42456224,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7306","reported_seq":7413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:55.236847+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:00:55.236847+0000","last_peered":"2024-09-11T05:00:55.236847+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:00:55.236847+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41800864,"num_objects":5103,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7505","reported_seq":7581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.765992+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:57.765992+0000","last_peered":"2024-09-11T05:00:57.765992+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:57.765992+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":678774408,"num_objects":82864,"num_object_clones":0,"num_object_copies":331456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82864,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112329,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1285799936,"data_stored":1285799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118545,"ondisk_log_size":118545,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745892,"kb_used_data":331028,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93625948,"statfs":{"total":96636764160,"available":95872970752,"internally_reserved":0,"allocated":338972672,"data_stored":338792703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":424799952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740624,"kb_used_data":334592,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93631216,"statfs":{"total":96636764160,"available":95878365184,"internally_reserved":0,"allocated":342622208,"data_stored":342442239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,1,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544976,"kb_used_data":260672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93826864,"statfs":{"total":96636764160,"available":96078708736,"internally_reserved":0,"allocated":266928128,"data_stored":266748159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291106514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739264,"kb_used_data":332784,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93632576,"statfs":{"total":96636764160,"available":95879757824,"internally_reserved":0,"allocated":340770816,"data_stored":340590847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,4,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339480576,"data_stored":339480576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266637312,"data_stored":266637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341856256,"data_stored":341856256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":337825792,"data_stored":337825792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:01.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:01.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7412", 'reported_seq': 7471, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:59.191514+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:00:59.191514+0000', 'last_peered': '2024-09-11T05:00:59.191514+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:00:59.191514+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7412, 'log_dups_size': 0, 'ondisk_log_size': 7412, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41795584, 'num_objects': 5102, 'num_object_clones': 0, 'num_object_copies': 20408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7412, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2796}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:01.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:01.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7507", 'reported_seq': 7590, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:57.594192+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:00:57.594192+0000', 'last_peered': '2024-09-11T05:00:57.594192+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:00:57.594192+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7507, 'log_dups_size': 0, 'ondisk_log_size': 7507, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42849440, 'num_objects': 5231, 'num_object_clones': 0, 'num_object_copies': 20924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5231, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7505, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:01.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:01.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7124", 'reported_seq': 7219, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:54.964839+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:00:54.964839+0000', 'last_peered': '2024-09-11T05:00:54.964839+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:00:54.964839+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7124, 'log_dups_size': 0, 'ondisk_log_size': 7124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41253480, 'num_objects': 5037, 'num_object_clones': 0, 'num_object_copies': 20148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5037, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7123, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2824}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:01.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:01.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7295", 'reported_seq': 7373, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:57.660705+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:00:57.660705+0000', 'last_peered': '2024-09-11T05:00:57.660705+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:00:57.660705+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7295, 'log_dups_size': 0, 'ondisk_log_size': 7295, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41948320, 'num_objects': 5121, 'num_object_clones': 0, 'num_object_copies': 20484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7293, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2848}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7353", 'reported_seq': 7430, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:59.398898+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:00:59.398898+0000', 'last_peered': '2024-09-11T05:00:59.398898+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:00:59.398898+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7353, 'log_dups_size': 0, 'ondisk_log_size': 7353, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42415264, 'num_objects': 5178, 'num_object_clones': 0, 'num_object_copies': 20712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2243, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7351, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2937}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7490", 'reported_seq': 7562, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:59.273244+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:00:59.273244+0000', 'last_peered': '2024-09-11T05:00:59.273244+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:00:59.273244+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7490, 'log_dups_size': 0, 'ondisk_log_size': 7490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42188800, 'num_objects': 5150, 'num_object_clones': 0, 'num_object_copies': 20600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5150, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7490, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2842}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7427", 'reported_seq': 7501, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:57.666869+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:00:57.666869+0000', 'last_peered': '2024-09-11T05:00:57.666869+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:00:57.666869+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7427, 'log_dups_size': 0, 'ondisk_log_size': 7427, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43439264, 'num_objects': 5303, 'num_object_clones': 0, 'num_object_copies': 21212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7425, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2989}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7564", 'reported_seq': 9376, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:57.739387+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:00:57.739387+0000', 'last_peered': '2024-09-11T05:00:57.739387+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:00:57.739387+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7564, 'log_dups_size': 0, 'ondisk_log_size': 7564, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43095200, 'num_objects': 5261, 'num_object_clones': 0, 'num_object_copies': 21044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1489, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7562, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 874, 'num_bytes_recovered': 7102464, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3772}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:01.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7606", 'reported_seq': 7693, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:59.363902+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:00:59.363902+0000', 'last_peered': '2024-09-11T05:00:59.363902+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:00:59.363902+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7606, 'log_dups_size': 0, 'ondisk_log_size': 7606, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42445120, 'num_objects': 5182, 'num_object_clones': 0, 'num_object_copies': 20728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2384, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7603, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2801}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7338", 'reported_seq': 7436, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:55.084116+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:00:55.084116+0000', 'last_peered': '2024-09-11T05:00:55.084116+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:00:55.084116+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7338, 'log_dups_size': 0, 'ondisk_log_size': 7338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42513568, 'num_objects': 5190, 'num_object_clones': 0, 'num_object_copies': 20760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7336, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2880}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7499", 'reported_seq': 7588, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:59.368736+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:00:59.368736+0000', 'last_peered': '2024-09-11T05:00:59.368736+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:00:59.368736+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7499, 'log_dups_size': 0, 'ondisk_log_size': 7499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42217200, 'num_objects': 5154, 'num_object_clones': 0, 'num_object_copies': 20616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7496, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2845}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7461", 'reported_seq': 7550, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:55.062908+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:00:55.062908+0000', 'last_peered': '2024-09-11T05:00:55.062908+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:00:55.062908+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7461, 'log_dups_size': 0, 'ondisk_log_size': 7461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43104848, 'num_objects': 5262, 'num_object_clones': 0, 'num_object_copies': 21048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2328, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7460, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2935}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:01.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7409", 'reported_seq': 7486, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:59.162794+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:00:59.162794+0000', 'last_peered': '2024-09-11T05:00:59.162794+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:00:59.162794+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7409, 'log_dups_size': 0, 'ondisk_log_size': 7409, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42456224, 'num_objects': 5183, 'num_object_clones': 0, 'num_object_copies': 20732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2278, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7407, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7306", 'reported_seq': 7413, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:55.236847+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:00:55.236847+0000', 'last_peered': '2024-09-11T05:00:55.236847+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:00:55.236847+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7306, 'log_dups_size': 0, 'ondisk_log_size': 7306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41800864, 'num_objects': 5103, 'num_object_clones': 0, 'num_object_copies': 20412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7304, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:01.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7505", 'reported_seq': 7581, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:00:57.765992+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:00:57.765992+0000', 'last_peered': '2024-09-11T05:00:57.765992+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:00:57.765992+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7505, 'log_dups_size': 0, 'ondisk_log_size': 7505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43185312, 'num_objects': 5272, 'num_object_clones': 0, 'num_object_copies': 21088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7503, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:01.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:01.828 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:01.829 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33654/331456 objects degraded (10.153%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":388,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82864,"data_bytes":678774408,"bytes_used":2837254144,"bytes_avail":383709802496,"bytes_total":386547056640,"degraded_objects":33654,"degraded_total":331456,"degraded_ratio":0.10153383857887623,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":116006,"recovering_keys_per_sec":0,"num_objects_recovered":191,"num_bytes_recovered":1392640,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":268},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:01.865 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:01.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:02.449 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:02.471 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-11T05:01:01.734732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":676472456,"num_objects":82583,"num_object_clones":0,"num_object_copies":330332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82583,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112610,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118826,"ondisk_log_size":118826,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770156,"kb_used_data":1254312,"kb_used_omap":14,"kb_used_meta":1515761,"kb_avail":374717204,"statfs":{"total":386547056640,"available":383710416896,"internally_reserved":0,"allocated":1284415488,"data_stored":1283695612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1552139830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,11,9,6,3,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21381120,"num_objects":-2610,"num_object_clones":0,"num_object_copies":-10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2610,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006107"},"pg_stats":[{"pgid":"1.f","version":"33'7412","reported_seq":7471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.191514+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:00:59.191514+0000","last_peered":"2024-09-11T05:00:59.191514+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:00:59.191514+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41795584,"num_objects":5102,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7507","reported_seq":7590,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.594192+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:00:57.594192+0000","last_peered":"2024-09-11T05:00:57.594192+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:00:57.594192+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7507,"log_dups_size":0,"ondisk_log_size":7507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42849440,"num_objects":5231,"num_object_clones":0,"num_object_copies":20924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7179","reported_seq":7274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:00.264728+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:00.264728+0000","last_peered":"2024-09-11T05:01:00.264728+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:00.264728+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7179,"log_dups_size":0,"ondisk_log_size":7179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40802920,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7178,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2769}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7295","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.660705+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:00:57.660705+0000","last_peered":"2024-09-11T05:00:57.660705+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:00:57.660705+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7295,"log_dups_size":0,"ondisk_log_size":7295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948320,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7293,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7353","reported_seq":7430,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.398898+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:00:59.398898+0000","last_peered":"2024-09-11T05:00:59.398898+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:00:59.398898+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7353,"log_dups_size":0,"ondisk_log_size":7353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42415264,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7490","reported_seq":7562,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.273244+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:00:59.273244+0000","last_peered":"2024-09-11T05:00:59.273244+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:00:59.273244+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42188800,"num_objects":5150,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7427","reported_seq":7501,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.666869+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:00:57.666869+0000","last_peered":"2024-09-11T05:00:57.666869+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:00:57.666869+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43439264,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7425,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2989}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7564","reported_seq":9376,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.739387+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:00:57.739387+0000","last_peered":"2024-09-11T05:00:57.739387+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:00:57.739387+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7564,"log_dups_size":0,"ondisk_log_size":7564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095200,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7606","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.363902+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:00:59.363902+0000","last_peered":"2024-09-11T05:00:59.363902+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:00:59.363902+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42445120,"num_objects":5182,"num_object_clones":0,"num_object_copies":20728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7603,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7400","reported_seq":7498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:00.262864+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:00.262864+0000","last_peered":"2024-09-11T05:01:00.262864+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:00.262864+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005664,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7398,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2818}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7499","reported_seq":7588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.368736+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:00:59.368736+0000","last_peered":"2024-09-11T05:00:59.368736+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:00:59.368736+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42217200,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7249","reported_seq":11844,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:00:58.704754+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:00:58.704754+0000","last_peered":"2024-09-11T05:00:58.704754+0000","last_clean":"2024-09-11T05:00:58.704754+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:00:58.704754+0000","last_undegraded":"2024-09-11T05:00:58.704754+0000","last_fullsized":"2024-09-11T05:00:58.704754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42065920,"num_objects":5135,"num_object_clones":0,"num_object_copies":20540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1059,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7542","reported_seq":7631,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:00.194641+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:00.194641+0000","last_peered":"2024-09-11T05:01:00.194641+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:00.194641+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42441296,"num_objects":5181,"num_object_clones":0,"num_object_copies":20724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7541,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7409","reported_seq":7486,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:59.162794+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:00:59.162794+0000","last_peered":"2024-09-11T05:00:59.162794+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:00:59.162794+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7409,"log_dups_size":0,"ondisk_log_size":7409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42456224,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7389","reported_seq":7496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:00.137165+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:00.137165+0000","last_peered":"2024-09-11T05:01:00.137165+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:00.137165+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41120928,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2714}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7505","reported_seq":7581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:00:57.765992+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:00:57.765992+0000","last_peered":"2024-09-11T05:00:57.765992+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:00:57.765992+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185312,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":676472456,"num_objects":82583,"num_object_clones":0,"num_object_copies":330332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82583,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112610,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":7102464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280913408,"data_stored":1280913408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118826,"ondisk_log_size":118826,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745892,"kb_used_data":331028,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93625948,"statfs":{"total":96636764160,"available":95872970752,"internally_reserved":0,"allocated":338972672,"data_stored":338792703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":424799952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740024,"kb_used_data":329828,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93631816,"statfs":{"total":96636764160,"available":95878979584,"internally_reserved":0,"allocated":337743872,"data_stored":337563903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544976,"kb_used_data":260672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93826864,"statfs":{"total":96636764160,"available":96078708736,"internally_reserved":0,"allocated":266928128,"data_stored":266748159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":291106514},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739264,"kb_used_data":332784,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93632576,"statfs":{"total":96636764160,"available":95879757824,"internally_reserved":0,"allocated":340770816,"data_stored":340590847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,4,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339480576,"data_stored":339480576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266637312,"data_stored":266637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336969728,"data_stored":336969728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":337825792,"data_stored":337825792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:05.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:06.140 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:06.166 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-11T05:01:05.737057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667379336,"num_objects":81473,"num_object_clones":0,"num_object_copies":325892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113720,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119936,"ondisk_log_size":119936,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2813432,"kb_used_data":1240668,"kb_used_omap":14,"kb_used_meta":1572657,"kb_avail":374673928,"statfs":{"total":386547056640,"available":383666102272,"internally_reserved":0,"allocated":1270444032,"data_stored":1269724156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1610401334},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,10,17,5,4,0,0,0,0,2,0,0,4],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25272320,"num_objects":-3085,"num_object_clones":0,"num_object_copies":-12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006623"},"pg_stats":[{"pgid":"1.f","version":"33'7487","reported_seq":7548,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.195627+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:04.195627+0000","last_peered":"2024-09-11T05:01:04.195627+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:04.195627+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7487,"log_dups_size":0,"ondisk_log_size":7487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41181184,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7580","reported_seq":7663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.802609+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:02.802609+0000","last_peered":"2024-09-11T05:01:02.802609+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:02.802609+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42251424,"num_objects":5158,"num_object_clones":0,"num_object_copies":20632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7236","reported_seq":7331,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.336120+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:05.336120+0000","last_peered":"2024-09-11T05:01:05.336120+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:05.336120+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7236,"log_dups_size":0,"ondisk_log_size":7236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40335976,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7235,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2712}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7370","reported_seq":7448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.478498+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:02.478498+0000","last_peered":"2024-09-11T05:01:02.478498+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:02.478498+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7370,"log_dups_size":0,"ondisk_log_size":7370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41333920,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7368,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7423","reported_seq":7504,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.365462+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:04.365462+0000","last_peered":"2024-09-11T05:01:04.365462+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:04.365462+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841824,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7563","reported_seq":7637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.331852+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:04.331852+0000","last_peered":"2024-09-11T05:01:04.331852+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:04.331852+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41590784,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7493","reported_seq":7567,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.511909+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:02.511909+0000","last_peered":"2024-09-11T05:01:02.511909+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:02.511909+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7493,"log_dups_size":0,"ondisk_log_size":7493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42898592,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7491,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2923}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7631","reported_seq":9602,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.815414+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:02.815414+0000","last_peered":"2024-09-11T05:01:02.815414+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:02.815414+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546336,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7629,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7678","reported_seq":7767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.340043+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:04.340043+0000","last_peered":"2024-09-11T05:01:04.340043+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:04.340043+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7678,"log_dups_size":0,"ondisk_log_size":7678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41855296,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7463","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.184361+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:05.184361+0000","last_peered":"2024-09-11T05:01:05.184361+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:05.184361+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41489568,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2755}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7570","reported_seq":7659,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.399676+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:04.399676+0000","last_peered":"2024-09-11T05:01:04.399676+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:04.399676+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7570,"log_dups_size":0,"ondisk_log_size":7570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41635568,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7567,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7306","reported_seq":11901,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:03.723193+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:03.723193+0000","last_peered":"2024-09-11T05:01:03.723193+0000","last_clean":"2024-09-11T05:01:03.723193+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:03.723193+0000","last_undegraded":"2024-09-11T05:01:03.723193+0000","last_fullsized":"2024-09-11T05:01:03.723193+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41598976,"num_objects":5078,"num_object_clones":0,"num_object_copies":20312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1116,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7619","reported_seq":7708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.345914+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:05.345914+0000","last_peered":"2024-09-11T05:01:05.345914+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:05.345914+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41810512,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7618,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2777}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7485","reported_seq":7564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.337877+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:04.337877+0000","last_peered":"2024-09-11T05:01:04.337877+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:04.337877+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7485,"log_dups_size":0,"ondisk_log_size":7485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41833632,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7483,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7469","reported_seq":7576,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.301798+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:05.301798+0000","last_peered":"2024-09-11T05:01:05.301798+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:05.301798+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465568,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7563","reported_seq":7641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.775868+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:02.775868+0000","last_peered":"2024-09-11T05:01:02.775868+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:02.775868+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7561,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":667379336,"num_objects":81473,"num_object_clones":0,"num_object_copies":325892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113720,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267027968,"data_stored":1267027968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119936,"ondisk_log_size":119936,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763696,"kb_used_data":326604,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93608144,"statfs":{"total":96636764160,"available":95854739456,"internally_reserved":0,"allocated":334442496,"data_stored":334262527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,2,4,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757864,"kb_used_data":325460,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93613976,"statfs":{"total":96636764160,"available":95860711424,"internally_reserved":0,"allocated":333271040,"data_stored":333091071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,7,2,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549020,"kb_used_data":260556,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93822820,"statfs":{"total":96636764160,"available":96074567680,"internally_reserved":0,"allocated":266809344,"data_stored":266629375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295366354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742852,"kb_used_data":328048,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93628988,"statfs":{"total":96636764160,"available":95876083712,"internally_reserved":0,"allocated":335921152,"data_stored":335741183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,1,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":334913536,"data_stored":334913536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266514432,"data_stored":266514432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332300288,"data_stored":332300288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":333299712,"data_stored":333299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:06.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:06.731 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:06.732 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:06.761 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-11T05:01:05.737057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667379336,"num_objects":81473,"num_object_clones":0,"num_object_copies":325892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113720,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119936,"ondisk_log_size":119936,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2813432,"kb_used_data":1240668,"kb_used_omap":14,"kb_used_meta":1572657,"kb_avail":374673928,"statfs":{"total":386547056640,"available":383666102272,"internally_reserved":0,"allocated":1270444032,"data_stored":1269724156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1610401334},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,10,17,5,4,0,0,0,0,2,0,0,4],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25272320,"num_objects":-3085,"num_object_clones":0,"num_object_copies":-12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006623"},"pg_stats":[{"pgid":"1.f","version":"33'7487","reported_seq":7548,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.195627+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:04.195627+0000","last_peered":"2024-09-11T05:01:04.195627+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:04.195627+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7487,"log_dups_size":0,"ondisk_log_size":7487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41181184,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7580","reported_seq":7663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.802609+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:02.802609+0000","last_peered":"2024-09-11T05:01:02.802609+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:02.802609+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42251424,"num_objects":5158,"num_object_clones":0,"num_object_copies":20632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7236","reported_seq":7331,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.336120+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:05.336120+0000","last_peered":"2024-09-11T05:01:05.336120+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:05.336120+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7236,"log_dups_size":0,"ondisk_log_size":7236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40335976,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7235,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2712}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7370","reported_seq":7448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.478498+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:02.478498+0000","last_peered":"2024-09-11T05:01:02.478498+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:02.478498+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7370,"log_dups_size":0,"ondisk_log_size":7370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41333920,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7368,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7423","reported_seq":7504,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.365462+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:04.365462+0000","last_peered":"2024-09-11T05:01:04.365462+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:04.365462+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841824,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7563","reported_seq":7637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.331852+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:04.331852+0000","last_peered":"2024-09-11T05:01:04.331852+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:04.331852+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41590784,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7493","reported_seq":7567,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.511909+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:02.511909+0000","last_peered":"2024-09-11T05:01:02.511909+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:02.511909+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7493,"log_dups_size":0,"ondisk_log_size":7493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42898592,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7491,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2923}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7631","reported_seq":9602,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.815414+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:02.815414+0000","last_peered":"2024-09-11T05:01:02.815414+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:02.815414+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546336,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7629,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7678","reported_seq":7767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.340043+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:04.340043+0000","last_peered":"2024-09-11T05:01:04.340043+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:04.340043+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7678,"log_dups_size":0,"ondisk_log_size":7678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41855296,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7463","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.184361+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:05.184361+0000","last_peered":"2024-09-11T05:01:05.184361+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:05.184361+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41489568,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2755}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7570","reported_seq":7659,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.399676+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:04.399676+0000","last_peered":"2024-09-11T05:01:04.399676+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:04.399676+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7570,"log_dups_size":0,"ondisk_log_size":7570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41635568,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7567,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7306","reported_seq":11901,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:03.723193+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:03.723193+0000","last_peered":"2024-09-11T05:01:03.723193+0000","last_clean":"2024-09-11T05:01:03.723193+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:03.723193+0000","last_undegraded":"2024-09-11T05:01:03.723193+0000","last_fullsized":"2024-09-11T05:01:03.723193+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41598976,"num_objects":5078,"num_object_clones":0,"num_object_copies":20312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1116,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7619","reported_seq":7708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.345914+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:05.345914+0000","last_peered":"2024-09-11T05:01:05.345914+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:05.345914+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41810512,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7618,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2777}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7485","reported_seq":7564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.337877+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:04.337877+0000","last_peered":"2024-09-11T05:01:04.337877+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:04.337877+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7485,"log_dups_size":0,"ondisk_log_size":7485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41833632,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7483,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7469","reported_seq":7576,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.301798+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:05.301798+0000","last_peered":"2024-09-11T05:01:05.301798+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:05.301798+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465568,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7563","reported_seq":7641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.775868+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:02.775868+0000","last_peered":"2024-09-11T05:01:02.775868+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:02.775868+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7561,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":667379336,"num_objects":81473,"num_object_clones":0,"num_object_copies":325892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113720,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267027968,"data_stored":1267027968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119936,"ondisk_log_size":119936,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763696,"kb_used_data":326604,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93608144,"statfs":{"total":96636764160,"available":95854739456,"internally_reserved":0,"allocated":334442496,"data_stored":334262527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,2,4,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757864,"kb_used_data":325460,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93613976,"statfs":{"total":96636764160,"available":95860711424,"internally_reserved":0,"allocated":333271040,"data_stored":333091071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,7,2,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549020,"kb_used_data":260556,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93822820,"statfs":{"total":96636764160,"available":96074567680,"internally_reserved":0,"allocated":266809344,"data_stored":266629375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295366354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742852,"kb_used_data":328048,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93628988,"statfs":{"total":96636764160,"available":95876083712,"internally_reserved":0,"allocated":335921152,"data_stored":335741183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,1,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":334913536,"data_stored":334913536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266514432,"data_stored":266514432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332300288,"data_stored":332300288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":333299712,"data_stored":333299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:06.872 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:06.873 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7487", 'reported_seq': 7548, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:04.195627+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:04.195627+0000', 'last_peered': '2024-09-11T05:01:04.195627+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:04.195627+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7487, 'log_dups_size': 0, 'ondisk_log_size': 7487, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41181184, 'num_objects': 5027, 'num_object_clones': 0, 'num_object_copies': 20108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7487, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2722}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:06.873 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:06.873 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7580", 'reported_seq': 7663, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:02.802609+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:02.802609+0000', 'last_peered': '2024-09-11T05:01:02.802609+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:02.802609+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7580, 'log_dups_size': 0, 'ondisk_log_size': 7580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42251424, 'num_objects': 5158, 'num_object_clones': 0, 'num_object_copies': 20632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7578, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:06.873 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:06.873 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7236", 'reported_seq': 7331, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:05.336120+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:05.336120+0000', 'last_peered': '2024-09-11T05:01:05.336120+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:05.336120+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7236, 'log_dups_size': 0, 'ondisk_log_size': 7236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40335976, 'num_objects': 4925, 'num_object_clones': 0, 'num_object_copies': 19700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4925, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7235, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2712}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7370", 'reported_seq': 7448, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:02.478498+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:02.478498+0000', 'last_peered': '2024-09-11T05:01:02.478498+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:02.478498+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7370, 'log_dups_size': 0, 'ondisk_log_size': 7370, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41333920, 'num_objects': 5046, 'num_object_clones': 0, 'num_object_copies': 20184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7368, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7423", 'reported_seq': 7504, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:04.365462+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:04.365462+0000', 'last_peered': '2024-09-11T05:01:04.365462+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:04.365462+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7423, 'log_dups_size': 0, 'ondisk_log_size': 7423, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41841824, 'num_objects': 5108, 'num_object_clones': 0, 'num_object_copies': 20432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7421, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2869}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7563", 'reported_seq': 7637, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:04.331852+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:04.331852+0000', 'last_peered': '2024-09-11T05:01:04.331852+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:04.331852+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7563, 'log_dups_size': 0, 'ondisk_log_size': 7563, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41590784, 'num_objects': 5077, 'num_object_clones': 0, 'num_object_copies': 20308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7563, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7493", 'reported_seq': 7567, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:02.511909+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:02.511909+0000', 'last_peered': '2024-09-11T05:01:02.511909+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:02.511909+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7493, 'log_dups_size': 0, 'ondisk_log_size': 7493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42898592, 'num_objects': 5237, 'num_object_clones': 0, 'num_object_copies': 20948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7491, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2923}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:06.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:06.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7631", 'reported_seq': 9602, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:02.815414+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:02.815414+0000', 'last_peered': '2024-09-11T05:01:02.815414+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:02.815414+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7631, 'log_dups_size': 0, 'ondisk_log_size': 7631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42546336, 'num_objects': 5194, 'num_object_clones': 0, 'num_object_copies': 20776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1411, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7629, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 954, 'num_bytes_recovered': 7741440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:06.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:06.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7678", 'reported_seq': 7767, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:04.340043+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:04.340043+0000', 'last_peered': '2024-09-11T05:01:04.340043+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:04.340043+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7678, 'log_dups_size': 0, 'ondisk_log_size': 7678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41855296, 'num_objects': 5110, 'num_object_clones': 0, 'num_object_copies': 20440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2383, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7675, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2730}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:06.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:06.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7463", 'reported_seq': 7561, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:05.184361+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:05.184361+0000', 'last_peered': '2024-09-11T05:01:05.184361+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:05.184361+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7463, 'log_dups_size': 0, 'ondisk_log_size': 7463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41489568, 'num_objects': 5065, 'num_object_clones': 0, 'num_object_copies': 20260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7461, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2755}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:06.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:06.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7570", 'reported_seq': 7659, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:04.399676+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:04.399676+0000', 'last_peered': '2024-09-11T05:01:04.399676+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:04.399676+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7570, 'log_dups_size': 0, 'ondisk_log_size': 7570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41635568, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7567, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2774}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:06.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:06.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7619", 'reported_seq': 7708, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:05.345914+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:05.345914+0000', 'last_peered': '2024-09-11T05:01:05.345914+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:05.345914+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7619, 'log_dups_size': 0, 'ondisk_log_size': 7619, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41810512, 'num_objects': 5104, 'num_object_clones': 0, 'num_object_copies': 20416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2328, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7618, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2777}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:06.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:06.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7485", 'reported_seq': 7564, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:04.337877+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:04.337877+0000', 'last_peered': '2024-09-11T05:01:04.337877+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:04.337877+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7485, 'log_dups_size': 0, 'ondisk_log_size': 7485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41833632, 'num_objects': 5107, 'num_object_clones': 0, 'num_object_copies': 20428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7483, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:06.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:06.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7469", 'reported_seq': 7576, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:05.301798+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:05.301798+0000', 'last_peered': '2024-09-11T05:01:05.301798+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:05.301798+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7469, 'log_dups_size': 0, 'ondisk_log_size': 7469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40465568, 'num_objects': 4940, 'num_object_clones': 0, 'num_object_copies': 19760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7467, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2634}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:06.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:06.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7563", 'reported_seq': 7641, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:02.775868+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:02.775868+0000', 'last_peered': '2024-09-11T05:01:02.775868+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:02.775868+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7563, 'log_dups_size': 0, 'ondisk_log_size': 7563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42710176, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7561, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2908}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:06.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:07.636 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:07.636 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33570/325892 objects degraded (10.301%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":394,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81473,"data_bytes":667379336,"bytes_used":2880954368,"bytes_avail":383666102272,"bytes_total":386547056640,"degraded_objects":33570,"degraded_total":325892,"degraded_ratio":0.10300958599781523,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":87333,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":1048576,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":256},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:07.662 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:07.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:08.279 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:08.280 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:08.299 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-11T05:01:07.737960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667379336,"num_objects":81473,"num_object_clones":0,"num_object_copies":325892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113720,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119936,"ondisk_log_size":119936,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2813432,"kb_used_data":1240668,"kb_used_omap":14,"kb_used_meta":1572657,"kb_avail":374673928,"statfs":{"total":386547056640,"available":383666102272,"internally_reserved":0,"allocated":1270444032,"data_stored":1269724156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1610401334},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,10,17,5,4,0,0,0,0,2,0,0,4],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18685952,"num_objects":-2281,"num_object_clones":0,"num_object_copies":-9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006696"},"pg_stats":[{"pgid":"1.f","version":"33'7487","reported_seq":7548,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.195627+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:04.195627+0000","last_peered":"2024-09-11T05:01:04.195627+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:04.195627+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7487,"log_dups_size":0,"ondisk_log_size":7487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41181184,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7580","reported_seq":7663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.802609+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:02.802609+0000","last_peered":"2024-09-11T05:01:02.802609+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:02.802609+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42251424,"num_objects":5158,"num_object_clones":0,"num_object_copies":20632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7236","reported_seq":7331,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.336120+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:05.336120+0000","last_peered":"2024-09-11T05:01:05.336120+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:05.336120+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7236,"log_dups_size":0,"ondisk_log_size":7236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40335976,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7235,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2712}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7370","reported_seq":7448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.478498+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:02.478498+0000","last_peered":"2024-09-11T05:01:02.478498+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:02.478498+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7370,"log_dups_size":0,"ondisk_log_size":7370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41333920,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7368,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7423","reported_seq":7504,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.365462+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:04.365462+0000","last_peered":"2024-09-11T05:01:04.365462+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:04.365462+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841824,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7563","reported_seq":7637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.331852+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:04.331852+0000","last_peered":"2024-09-11T05:01:04.331852+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:04.331852+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41590784,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7493","reported_seq":7567,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.511909+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:02.511909+0000","last_peered":"2024-09-11T05:01:02.511909+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:02.511909+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7493,"log_dups_size":0,"ondisk_log_size":7493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42898592,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7491,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2923}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7631","reported_seq":9602,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.815414+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:02.815414+0000","last_peered":"2024-09-11T05:01:02.815414+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:02.815414+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546336,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7629,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7678","reported_seq":7767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.340043+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:04.340043+0000","last_peered":"2024-09-11T05:01:04.340043+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:04.340043+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7678,"log_dups_size":0,"ondisk_log_size":7678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41855296,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7463","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.184361+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:05.184361+0000","last_peered":"2024-09-11T05:01:05.184361+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:05.184361+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41489568,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2755}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7570","reported_seq":7659,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.399676+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:04.399676+0000","last_peered":"2024-09-11T05:01:04.399676+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:04.399676+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7570,"log_dups_size":0,"ondisk_log_size":7570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41635568,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7567,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7306","reported_seq":11901,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:03.723193+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:03.723193+0000","last_peered":"2024-09-11T05:01:03.723193+0000","last_clean":"2024-09-11T05:01:03.723193+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:03.723193+0000","last_undegraded":"2024-09-11T05:01:03.723193+0000","last_fullsized":"2024-09-11T05:01:03.723193+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41598976,"num_objects":5078,"num_object_clones":0,"num_object_copies":20312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1116,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7619","reported_seq":7708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.345914+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:05.345914+0000","last_peered":"2024-09-11T05:01:05.345914+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:05.345914+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41810512,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7618,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2777}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7485","reported_seq":7564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:04.337877+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:04.337877+0000","last_peered":"2024-09-11T05:01:04.337877+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:04.337877+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7485,"log_dups_size":0,"ondisk_log_size":7485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41833632,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7483,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7469","reported_seq":7576,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:05.301798+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:05.301798+0000","last_peered":"2024-09-11T05:01:05.301798+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:05.301798+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40465568,"num_objects":4940,"num_object_clones":0,"num_object_copies":19760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7563","reported_seq":7641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:02.775868+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:02.775868+0000","last_peered":"2024-09-11T05:01:02.775868+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:02.775868+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710176,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7561,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":667379336,"num_objects":81473,"num_object_clones":0,"num_object_copies":325892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113720,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":7741440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267027968,"data_stored":1267027968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119936,"ondisk_log_size":119936,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763696,"kb_used_data":326604,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93608144,"statfs":{"total":96636764160,"available":95854739456,"internally_reserved":0,"allocated":334442496,"data_stored":334262527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,2,4,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757864,"kb_used_data":325460,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93613976,"statfs":{"total":96636764160,"available":95860711424,"internally_reserved":0,"allocated":333271040,"data_stored":333091071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,7,2,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549020,"kb_used_data":260556,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93822820,"statfs":{"total":96636764160,"available":96074567680,"internally_reserved":0,"allocated":266809344,"data_stored":266629375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":295366354},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742852,"kb_used_data":328048,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93628988,"statfs":{"total":96636764160,"available":95876083712,"internally_reserved":0,"allocated":335921152,"data_stored":335741183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,1,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":334913536,"data_stored":334913536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266514432,"data_stored":266514432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332300288,"data_stored":332300288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":333299712,"data_stored":333299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:11.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:11.919 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:11.920 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:11.942 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-11T05:01:11.739606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657704584,"num_objects":80292,"num_object_clones":0,"num_object_copies":321168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80292,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114901,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1152,"num_bytes_recovered":8118272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121117,"ondisk_log_size":121117,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816220,"kb_used_data":1225424,"kb_used_omap":14,"kb_used_meta":1590705,"kb_avail":374671140,"statfs":{"total":386547056640,"available":383663247360,"internally_reserved":0,"allocated":1254834176,"data_stored":1254114300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1628882486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,3,13,10,16,3,0,0,0,0,0,0,0,4],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21069824,"num_objects":-2572,"num_object_clones":0,"num_object_copies":-10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2572,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1015808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006938"},"pg_stats":[{"pgid":"1.f","version":"33'7558","reported_seq":7619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.408432+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:09.408432+0000","last_peered":"2024-09-11T05:01:09.408432+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:09.408432+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7558,"log_dups_size":0,"ondisk_log_size":7558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40599552,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7649","reported_seq":7735,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.838523+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:07.838523+0000","last_peered":"2024-09-11T05:01:07.838523+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:07.838523+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41686176,"num_objects":5089,"num_object_clones":0,"num_object_copies":20356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7647,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7312","reported_seq":7407,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.367736+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:10.367736+0000","last_peered":"2024-09-11T05:01:10.367736+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:10.367736+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39713384,"num_objects":4849,"num_object_clones":0,"num_object_copies":19396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7311,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2636}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7454","reported_seq":7532,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.733149+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:07.733149+0000","last_peered":"2024-09-11T05:01:07.733149+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:07.733149+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40645792,"num_objects":4962,"num_object_clones":0,"num_object_copies":19848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7452,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7472","reported_seq":7553,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.338952+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:09.338952+0000","last_peered":"2024-09-11T05:01:09.338952+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:09.338952+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7472,"log_dups_size":0,"ondisk_log_size":7472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41440416,"num_objects":5059,"num_object_clones":0,"num_object_copies":20236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2820}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7645","reported_seq":7719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.409979+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:09.409979+0000","last_peered":"2024-09-11T05:01:09.409979+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:09.409979+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7645,"log_dups_size":0,"ondisk_log_size":7645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40919040,"num_objects":4995,"num_object_clones":0,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7568","reported_seq":7644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.797336+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:07.797336+0000","last_peered":"2024-09-11T05:01:07.797336+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:07.797336+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284192,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2849}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7711","reported_seq":9775,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.846399+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:07.846399+0000","last_peered":"2024-09-11T05:01:07.846399+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:07.846399+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7711,"log_dups_size":0,"ondisk_log_size":7711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41890976,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7709,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":8118272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7757","reported_seq":7846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.362323+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:09.362323+0000","last_peered":"2024-09-11T05:01:09.362323+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:09.362323+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41208128,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7550","reported_seq":7648,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.369747+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:10.369747+0000","last_peered":"2024-09-11T05:01:10.369747+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:10.369747+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7550,"log_dups_size":0,"ondisk_log_size":7550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40776864,"num_objects":4978,"num_object_clones":0,"num_object_copies":19912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2668}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7646","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.457633+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:09.457633+0000","last_peered":"2024-09-11T05:01:09.457633+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:09.457633+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41012976,"num_objects":5007,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7643,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7373","reported_seq":11967,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:08.856695+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:08.856695+0000","last_peered":"2024-09-11T05:01:08.856695+0000","last_clean":"2024-09-11T05:01:08.856695+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:08.856695+0000","last_undegraded":"2024-09-11T05:01:08.856695+0000","last_fullsized":"2024-09-11T05:01:08.856695+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7373,"log_dups_size":0,"ondisk_log_size":7373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41050112,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7684","reported_seq":7772,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.218593+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:10.218593+0000","last_peered":"2024-09-11T05:01:10.218593+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:10.218593+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7684,"log_dups_size":0,"ondisk_log_size":7684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41278032,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7683,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7562","reported_seq":7641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.316359+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:09.316359+0000","last_peered":"2024-09-11T05:01:09.316359+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:09.316359+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41202848,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2755}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7529","reported_seq":7636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.344505+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:10.344505+0000","last_peered":"2024-09-11T05:01:10.344505+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:10.344505+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7647","reported_seq":7725,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.755908+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:07.755908+0000","last_peered":"2024-09-11T05:01:07.755908+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:07.755908+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42022048,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":657704584,"num_objects":80292,"num_object_clones":0,"num_object_copies":321168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80292,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114901,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1152,"num_bytes_recovered":8118272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1252237312,"data_stored":1252237312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121117,"ondisk_log_size":121117,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746440,"kb_used_data":321844,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93625400,"statfs":{"total":96636764160,"available":95872409600,"internally_reserved":0,"allocated":329568256,"data_stored":329388287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757288,"kb_used_data":320728,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93614552,"statfs":{"total":96636764160,"available":95861301248,"internally_reserved":0,"allocated":328425472,"data_stored":328245503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,0,3,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552808,"kb_used_data":260180,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93819032,"statfs":{"total":96636764160,"available":96070688768,"internally_reserved":0,"allocated":266424320,"data_stored":266244351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299626194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759684,"kb_used_data":322672,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93612156,"statfs":{"total":96636764160,"available":95858847744,"internally_reserved":0,"allocated":330416128,"data_stored":330236159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,3,1,8,1,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":330035200,"data_stored":330035200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266137600,"data_stored":266137600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327573504,"data_stored":327573504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328491008,"data_stored":328491008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:11.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:12.615 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-11T05:01:11.739606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657704584,"num_objects":80292,"num_object_clones":0,"num_object_copies":321168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80292,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114901,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1152,"num_bytes_recovered":8118272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121117,"ondisk_log_size":121117,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816220,"kb_used_data":1225424,"kb_used_omap":14,"kb_used_meta":1590705,"kb_avail":374671140,"statfs":{"total":386547056640,"available":383663247360,"internally_reserved":0,"allocated":1254834176,"data_stored":1254114300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1628882486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,3,13,10,16,3,0,0,0,0,0,0,0,4],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21069824,"num_objects":-2572,"num_object_clones":0,"num_object_copies":-10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2572,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1015808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006938"},"pg_stats":[{"pgid":"1.f","version":"33'7558","reported_seq":7619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.408432+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:09.408432+0000","last_peered":"2024-09-11T05:01:09.408432+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:09.408432+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7558,"log_dups_size":0,"ondisk_log_size":7558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40599552,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7649","reported_seq":7735,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.838523+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:07.838523+0000","last_peered":"2024-09-11T05:01:07.838523+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:07.838523+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41686176,"num_objects":5089,"num_object_clones":0,"num_object_copies":20356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7647,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7312","reported_seq":7407,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.367736+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:10.367736+0000","last_peered":"2024-09-11T05:01:10.367736+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:10.367736+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39713384,"num_objects":4849,"num_object_clones":0,"num_object_copies":19396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7311,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2636}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7454","reported_seq":7532,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.733149+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:07.733149+0000","last_peered":"2024-09-11T05:01:07.733149+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:07.733149+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40645792,"num_objects":4962,"num_object_clones":0,"num_object_copies":19848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7452,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7472","reported_seq":7553,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.338952+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:09.338952+0000","last_peered":"2024-09-11T05:01:09.338952+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:09.338952+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7472,"log_dups_size":0,"ondisk_log_size":7472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41440416,"num_objects":5059,"num_object_clones":0,"num_object_copies":20236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2820}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7645","reported_seq":7719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.409979+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:09.409979+0000","last_peered":"2024-09-11T05:01:09.409979+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:09.409979+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7645,"log_dups_size":0,"ondisk_log_size":7645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40919040,"num_objects":4995,"num_object_clones":0,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7568","reported_seq":7644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.797336+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:07.797336+0000","last_peered":"2024-09-11T05:01:07.797336+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:07.797336+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42284192,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2849}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7711","reported_seq":9775,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.846399+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:07.846399+0000","last_peered":"2024-09-11T05:01:07.846399+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:07.846399+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7711,"log_dups_size":0,"ondisk_log_size":7711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41890976,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7709,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":8118272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7757","reported_seq":7846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.362323+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:09.362323+0000","last_peered":"2024-09-11T05:01:09.362323+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:09.362323+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41208128,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7550","reported_seq":7648,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.369747+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:10.369747+0000","last_peered":"2024-09-11T05:01:10.369747+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:10.369747+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7550,"log_dups_size":0,"ondisk_log_size":7550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40776864,"num_objects":4978,"num_object_clones":0,"num_object_copies":19912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2668}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7646","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.457633+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:09.457633+0000","last_peered":"2024-09-11T05:01:09.457633+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:09.457633+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41012976,"num_objects":5007,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7643,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7373","reported_seq":11967,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:08.856695+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:08.856695+0000","last_peered":"2024-09-11T05:01:08.856695+0000","last_clean":"2024-09-11T05:01:08.856695+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:08.856695+0000","last_undegraded":"2024-09-11T05:01:08.856695+0000","last_fullsized":"2024-09-11T05:01:08.856695+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7373,"log_dups_size":0,"ondisk_log_size":7373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41050112,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7684","reported_seq":7772,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.218593+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:10.218593+0000","last_peered":"2024-09-11T05:01:10.218593+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:10.218593+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7684,"log_dups_size":0,"ondisk_log_size":7684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41278032,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7683,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7562","reported_seq":7641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.316359+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:09.316359+0000","last_peered":"2024-09-11T05:01:09.316359+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:09.316359+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41202848,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2755}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7529","reported_seq":7636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.344505+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:10.344505+0000","last_peered":"2024-09-11T05:01:10.344505+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:10.344505+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7647","reported_seq":7725,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:07.755908+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:07.755908+0000","last_peered":"2024-09-11T05:01:07.755908+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:07.755908+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42022048,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":657704584,"num_objects":80292,"num_object_clones":0,"num_object_copies":321168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80292,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114901,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1152,"num_bytes_recovered":8118272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1252237312,"data_stored":1252237312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121117,"ondisk_log_size":121117,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746440,"kb_used_data":321844,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93625400,"statfs":{"total":96636764160,"available":95872409600,"internally_reserved":0,"allocated":329568256,"data_stored":329388287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757288,"kb_used_data":320728,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93614552,"statfs":{"total":96636764160,"available":95861301248,"internally_reserved":0,"allocated":328425472,"data_stored":328245503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,0,3,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552808,"kb_used_data":260180,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93819032,"statfs":{"total":96636764160,"available":96070688768,"internally_reserved":0,"allocated":266424320,"data_stored":266244351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299626194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759684,"kb_used_data":322672,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93612156,"statfs":{"total":96636764160,"available":95858847744,"internally_reserved":0,"allocated":330416128,"data_stored":330236159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,3,1,8,1,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":330035200,"data_stored":330035200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266137600,"data_stored":266137600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327573504,"data_stored":327573504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328491008,"data_stored":328491008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:12.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:12.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7558", 'reported_seq': 7619, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:09.408432+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:09.408432+0000', 'last_peered': '2024-09-11T05:01:09.408432+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:09.408432+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7558, 'log_dups_size': 0, 'ondisk_log_size': 7558, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40599552, 'num_objects': 4956, 'num_object_clones': 0, 'num_object_copies': 19824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7558, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2651}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7649", 'reported_seq': 7735, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:07.838523+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:07.838523+0000', 'last_peered': '2024-09-11T05:01:07.838523+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:07.838523+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7649, 'log_dups_size': 0, 'ondisk_log_size': 7649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41686176, 'num_objects': 5089, 'num_object_clones': 0, 'num_object_copies': 20356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2271, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7647, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2820}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7312", 'reported_seq': 7407, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:10.367736+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:10.367736+0000', 'last_peered': '2024-09-11T05:01:10.367736+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:10.367736+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7312, 'log_dups_size': 0, 'ondisk_log_size': 7312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39713384, 'num_objects': 4849, 'num_object_clones': 0, 'num_object_copies': 19396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4849, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7311, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2636}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7454", 'reported_seq': 7532, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:07.733149+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:07.733149+0000', 'last_peered': '2024-09-11T05:01:07.733149+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:07.733149+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7454, 'log_dups_size': 0, 'ondisk_log_size': 7454, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40645792, 'num_objects': 4962, 'num_object_clones': 0, 'num_object_copies': 19848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4962, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7452, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2689}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7472", 'reported_seq': 7553, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:09.338952+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:09.338952+0000', 'last_peered': '2024-09-11T05:01:09.338952+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:09.338952+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7472, 'log_dups_size': 0, 'ondisk_log_size': 7472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41440416, 'num_objects': 5059, 'num_object_clones': 0, 'num_object_copies': 20236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7470, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2820}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:12.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7645", 'reported_seq': 7719, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:09.409979+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:09.409979+0000', 'last_peered': '2024-09-11T05:01:09.409979+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:09.409979+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7645, 'log_dups_size': 0, 'ondisk_log_size': 7645, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40919040, 'num_objects': 4995, 'num_object_clones': 0, 'num_object_copies': 19980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7645, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2688}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7568", 'reported_seq': 7644, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:07.797336+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:07.797336+0000', 'last_peered': '2024-09-11T05:01:07.797336+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:07.797336+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7568, 'log_dups_size': 0, 'ondisk_log_size': 7568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42284192, 'num_objects': 5162, 'num_object_clones': 0, 'num_object_copies': 20648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7566, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2849}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7711", 'reported_seq': 9775, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:07.846399+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:07.846399+0000', 'last_peered': '2024-09-11T05:01:07.846399+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:07.846399+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7711, 'log_dups_size': 0, 'ondisk_log_size': 7711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41890976, 'num_objects': 5114, 'num_object_clones': 0, 'num_object_copies': 20456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1365, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7709, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1000, 'num_bytes_recovered': 8118272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7757", 'reported_seq': 7846, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:09.362323+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:09.362323+0000', 'last_peered': '2024-09-11T05:01:09.362323+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:09.362323+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7757, 'log_dups_size': 0, 'ondisk_log_size': 7757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41208128, 'num_objects': 5031, 'num_object_clones': 0, 'num_object_copies': 20124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2383, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7754, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2651}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7550", 'reported_seq': 7648, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:10.369747+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:10.369747+0000', 'last_peered': '2024-09-11T05:01:10.369747+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:10.369747+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7550, 'log_dups_size': 0, 'ondisk_log_size': 7550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40776864, 'num_objects': 4978, 'num_object_clones': 0, 'num_object_copies': 19912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7548, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2668}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:12.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7646", 'reported_seq': 7734, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:09.457633+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:09.457633+0000', 'last_peered': '2024-09-11T05:01:09.457633+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:09.457633+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7646, 'log_dups_size': 0, 'ondisk_log_size': 7646, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41012976, 'num_objects': 5007, 'num_object_clones': 0, 'num_object_copies': 20028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7643, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2698}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:12.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:12.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7684", 'reported_seq': 7772, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:10.218593+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:10.218593+0000', 'last_peered': '2024-09-11T05:01:10.218593+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:10.218593+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7684, 'log_dups_size': 0, 'ondisk_log_size': 7684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41278032, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 20156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2328, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7683, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:12.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:12.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7562", 'reported_seq': 7641, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:09.316359+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:09.316359+0000', 'last_peered': '2024-09-11T05:01:09.316359+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:09.316359+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7562, 'log_dups_size': 0, 'ondisk_log_size': 7562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41202848, 'num_objects': 5030, 'num_object_clones': 0, 'num_object_copies': 20120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7560, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2755}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:12.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:12.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7529", 'reported_seq': 7636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:10.344505+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:10.344505+0000', 'last_peered': '2024-09-11T05:01:10.344505+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:10.344505+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7529, 'log_dups_size': 0, 'ondisk_log_size': 7529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39974048, 'num_objects': 4880, 'num_object_clones': 0, 'num_object_copies': 19520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7527, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2574}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:12.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:12.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7647", 'reported_seq': 7725, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:07.755908+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:07.755908+0000', 'last_peered': '2024-09-11T05:01:07.755908+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:07.755908+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7647, 'log_dups_size': 0, 'ondisk_log_size': 7647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42022048, 'num_objects': 5130, 'num_object_clones': 0, 'num_object_copies': 20520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7645, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:12.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:13.451 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:13.451 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33521/321168 objects degraded (10.437%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":400,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80292,"data_bytes":657704584,"bytes_used":2883809280,"bytes_avail":383663247360,"bytes_total":386547056640,"degraded_objects":33521,"degraded_total":321168,"degraded_ratio":0.10437216659193942,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":84601,"recovering_keys_per_sec":0,"num_objects_recovered":135,"num_bytes_recovered":1015808,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":214},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:13.480 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:13.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:14.087 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:14.108 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-11T05:01:13.740473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654632584,"num_objects":79917,"num_object_clones":0,"num_object_copies":319668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115276,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121492,"ondisk_log_size":121492,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799220,"kb_used_data":1220908,"kb_used_omap":14,"kb_used_meta":1578225,"kb_avail":374688140,"statfs":{"total":386547056640,"available":383680655360,"internally_reserved":0,"allocated":1250209792,"data_stored":1249489916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1616102966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,10,10,10,3,0,0,0,0,0,5,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21839872,"num_objects":-2666,"num_object_clones":0,"num_object_copies":-10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005740"},"pg_stats":[{"pgid":"1.f","version":"33'7558","reported_seq":7619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.408432+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:09.408432+0000","last_peered":"2024-09-11T05:01:09.408432+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:09.408432+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7558,"log_dups_size":0,"ondisk_log_size":7558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40599552,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7718","reported_seq":7805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.887789+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:12.887789+0000","last_peered":"2024-09-11T05:01:12.887789+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:12.887789+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7718,"log_dups_size":0,"ondisk_log_size":7718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41120928,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7716,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7312","reported_seq":7407,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.367736+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:10.367736+0000","last_peered":"2024-09-11T05:01:10.367736+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:10.367736+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39713384,"num_objects":4849,"num_object_clones":0,"num_object_copies":19396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7311,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2636}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7523","reported_seq":7601,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.833667+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:12.833667+0000","last_peered":"2024-09-11T05:01:12.833667+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:12.833667+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40080544,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7521,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7472","reported_seq":7553,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.338952+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:09.338952+0000","last_peered":"2024-09-11T05:01:09.338952+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:09.338952+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7472,"log_dups_size":0,"ondisk_log_size":7472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41440416,"num_objects":5059,"num_object_clones":0,"num_object_copies":20236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2820}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7645","reported_seq":7719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.409979+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:09.409979+0000","last_peered":"2024-09-11T05:01:09.409979+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:09.409979+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7645,"log_dups_size":0,"ondisk_log_size":7645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40919040,"num_objects":4995,"num_object_clones":0,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7654","reported_seq":7730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.857082+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:12.857082+0000","last_peered":"2024-09-11T05:01:12.857082+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:12.857082+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7654,"log_dups_size":0,"ondisk_log_size":7654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41579680,"num_objects":5076,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7652,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2763}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7787","reported_seq":9963,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.908088+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:12.908088+0000","last_peered":"2024-09-11T05:01:12.908088+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:12.908088+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268384,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7785,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1056,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7757","reported_seq":7846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.362323+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:09.362323+0000","last_peered":"2024-09-11T05:01:09.362323+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:09.362323+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41208128,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7550","reported_seq":7648,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.369747+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:10.369747+0000","last_peered":"2024-09-11T05:01:10.369747+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:10.369747+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7550,"log_dups_size":0,"ondisk_log_size":7550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40776864,"num_objects":4978,"num_object_clones":0,"num_object_copies":19912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2668}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7646","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.457633+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:09.457633+0000","last_peered":"2024-09-11T05:01:09.457633+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:09.457633+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41012976,"num_objects":5007,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7643,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7373","reported_seq":11967,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:08.856695+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:08.856695+0000","last_peered":"2024-09-11T05:01:08.856695+0000","last_clean":"2024-09-11T05:01:08.856695+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:08.856695+0000","last_undegraded":"2024-09-11T05:01:08.856695+0000","last_fullsized":"2024-09-11T05:01:08.856695+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7373,"log_dups_size":0,"ondisk_log_size":7373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41050112,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7684","reported_seq":7772,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.218593+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:10.218593+0000","last_peered":"2024-09-11T05:01:10.218593+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:10.218593+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7684,"log_dups_size":0,"ondisk_log_size":7684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41278032,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7683,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7562","reported_seq":7641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:09.316359+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:09.316359+0000","last_peered":"2024-09-11T05:01:09.316359+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:09.316359+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41202848,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2755}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7529","reported_seq":7636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:10.344505+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:10.344505+0000","last_peered":"2024-09-11T05:01:10.344505+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:10.344505+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7527,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7722","reported_seq":7800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.906239+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:12.906239+0000","last_peered":"2024-09-11T05:01:12.906239+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:12.906239+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41407648,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":654632584,"num_objects":79917,"num_object_clones":0,"num_object_copies":319668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115276,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1247469568,"data_stored":1247469568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121492,"ondisk_log_size":121492,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746440,"kb_used_data":321844,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93625400,"statfs":{"total":96636764160,"available":95872409600,"internally_reserved":0,"allocated":329568256,"data_stored":329388287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757288,"kb_used_data":320728,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93614552,"statfs":{"total":96636764160,"available":95861301248,"internally_reserved":0,"allocated":328425472,"data_stored":328245503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,0,3,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552808,"kb_used_data":260180,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93819032,"statfs":{"total":96636764160,"available":96070688768,"internally_reserved":0,"allocated":266424320,"data_stored":266244351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":299626194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742684,"kb_used_data":318156,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93629156,"statfs":{"total":96636764160,"available":95876255744,"internally_reserved":0,"allocated":325791744,"data_stored":325611775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,2,1,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325267456,"data_stored":325267456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266137600,"data_stored":266137600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327573504,"data_stored":327573504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328491008,"data_stored":328491008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:17.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:17.751 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:17.773 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-11T05:01:17.742240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648210056,"num_objects":79133,"num_object_clones":0,"num_object_copies":316532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116060,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122276,"ondisk_log_size":122276,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2768788,"kb_used_data":1211272,"kb_used_omap":14,"kb_used_meta":1557425,"kb_avail":374718572,"statfs":{"total":386547056640,"available":383711817728,"internally_reserved":0,"allocated":1240342528,"data_stored":1239622652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1594803766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,5,5,5,4,1,0,0,0,0,0,5,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19169280,"num_objects":-2340,"num_object_clones":0,"num_object_copies":-9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005182"},"pg_stats":[{"pgid":"1.f","version":"33'7627","reported_seq":7690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.368616+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:14.368616+0000","last_peered":"2024-09-11T05:01:14.368616+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:14.368616+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40034304,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7718","reported_seq":7805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.887789+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:12.887789+0000","last_peered":"2024-09-11T05:01:12.887789+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:12.887789+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7718,"log_dups_size":0,"ondisk_log_size":7718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41120928,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7716,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7407","reported_seq":7502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.363551+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:15.363551+0000","last_peered":"2024-09-11T05:01:15.363551+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:15.363551+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7407,"log_dups_size":0,"ondisk_log_size":7407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38935144,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7406,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2541}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7523","reported_seq":7601,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.833667+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:12.833667+0000","last_peered":"2024-09-11T05:01:12.833667+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:12.833667+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40080544,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7521,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7541","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.398022+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:14.398022+0000","last_peered":"2024-09-11T05:01:14.398022+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:14.398022+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7541,"log_dups_size":0,"ondisk_log_size":7541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40875168,"num_objects":4990,"num_object_clones":0,"num_object_copies":19960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7539,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7709","reported_seq":7791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.427129+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:14.427129+0000","last_peered":"2024-09-11T05:01:14.427129+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:14.427129+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7709,"log_dups_size":0,"ondisk_log_size":7709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40394752,"num_objects":4931,"num_object_clones":0,"num_object_copies":19724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7709,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7654","reported_seq":7730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.857082+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:12.857082+0000","last_peered":"2024-09-11T05:01:12.857082+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:12.857082+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7654,"log_dups_size":0,"ondisk_log_size":7654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41579680,"num_objects":5076,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7652,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2763}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7787","reported_seq":9963,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.908088+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:12.908088+0000","last_peered":"2024-09-11T05:01:12.908088+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:12.908088+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268384,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7785,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1056,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7832","reported_seq":7925,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.372834+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:14.372834+0000","last_peered":"2024-09-11T05:01:14.372834+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:14.372834+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40593728,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7829,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7615","reported_seq":7715,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.389206+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:15.389206+0000","last_peered":"2024-09-11T05:01:15.389206+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:15.389206+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244384,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7700","reported_seq":7789,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.494844+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:14.494844+0000","last_peered":"2024-09-11T05:01:14.494844+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:14.494844+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40570608,"num_objects":4953,"num_object_clones":0,"num_object_copies":19812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7455","reported_seq":12050,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:13.831312+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:13.831312+0000","last_peered":"2024-09-11T05:01:13.831312+0000","last_clean":"2024-09-11T05:01:13.831312+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:13.831312+0000","last_undegraded":"2024-09-11T05:01:13.831312+0000","last_fullsized":"2024-09-11T05:01:13.831312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40378368,"num_objects":4929,"num_object_clones":0,"num_object_copies":19716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1265,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7759","reported_seq":7850,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.256101+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:15.256101+0000","last_peered":"2024-09-11T05:01:15.256101+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:15.256101+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40663632,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2638}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7641","reported_seq":7720,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.464121+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:14.464121+0000","last_peered":"2024-09-11T05:01:14.464121+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:14.464121+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7641,"log_dups_size":0,"ondisk_log_size":7641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40555680,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7639,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2676}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7586","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.327570+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:15.327570+0000","last_peered":"2024-09-11T05:01:15.327570+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:15.327570+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39507104,"num_objects":4823,"num_object_clones":0,"num_object_copies":19292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2517}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7722","reported_seq":7800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.906239+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:12.906239+0000","last_peered":"2024-09-11T05:01:12.906239+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:12.906239+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41407648,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":648210056,"num_objects":79133,"num_object_clones":0,"num_object_copies":316532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116060,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1237762048,"data_stored":1237762048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122276,"ondisk_log_size":122276,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746016,"kb_used_data":317256,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93625824,"statfs":{"total":96636764160,"available":95872843776,"internally_reserved":0,"allocated":324870144,"data_stored":324690175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740072,"kb_used_data":315992,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93631768,"statfs":{"total":96636764160,"available":95878930432,"internally_reserved":0,"allocated":323575808,"data_stored":323395839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,1,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540016,"kb_used_data":259868,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93831824,"statfs":{"total":96636764160,"available":96083787776,"internally_reserved":0,"allocated":266104832,"data_stored":265924863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286846674},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742684,"kb_used_data":318156,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93629156,"statfs":{"total":96636764160,"available":95876255744,"internally_reserved":0,"allocated":325791744,"data_stored":325611775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,2,1,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325267456,"data_stored":325267456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265826304,"data_stored":265826304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322842624,"data_stored":322842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323825664,"data_stored":323825664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:17.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:18.414 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:18.414 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:18.439 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-11T05:01:17.742240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648210056,"num_objects":79133,"num_object_clones":0,"num_object_copies":316532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116060,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122276,"ondisk_log_size":122276,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2768788,"kb_used_data":1211272,"kb_used_omap":14,"kb_used_meta":1557425,"kb_avail":374718572,"statfs":{"total":386547056640,"available":383711817728,"internally_reserved":0,"allocated":1240342528,"data_stored":1239622652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1594803766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,5,5,5,4,1,0,0,0,0,0,5,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19169280,"num_objects":-2340,"num_object_clones":0,"num_object_copies":-9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005182"},"pg_stats":[{"pgid":"1.f","version":"33'7627","reported_seq":7690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.368616+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:14.368616+0000","last_peered":"2024-09-11T05:01:14.368616+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:14.368616+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40034304,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7718","reported_seq":7805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.887789+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:12.887789+0000","last_peered":"2024-09-11T05:01:12.887789+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:12.887789+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7718,"log_dups_size":0,"ondisk_log_size":7718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41120928,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7716,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7407","reported_seq":7502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.363551+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:15.363551+0000","last_peered":"2024-09-11T05:01:15.363551+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:15.363551+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7407,"log_dups_size":0,"ondisk_log_size":7407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38935144,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7406,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2541}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7523","reported_seq":7601,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.833667+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:12.833667+0000","last_peered":"2024-09-11T05:01:12.833667+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:12.833667+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40080544,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7521,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7541","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.398022+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:14.398022+0000","last_peered":"2024-09-11T05:01:14.398022+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:14.398022+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7541,"log_dups_size":0,"ondisk_log_size":7541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40875168,"num_objects":4990,"num_object_clones":0,"num_object_copies":19960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7539,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7709","reported_seq":7791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.427129+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:14.427129+0000","last_peered":"2024-09-11T05:01:14.427129+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:14.427129+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7709,"log_dups_size":0,"ondisk_log_size":7709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40394752,"num_objects":4931,"num_object_clones":0,"num_object_copies":19724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7709,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7654","reported_seq":7730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.857082+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:12.857082+0000","last_peered":"2024-09-11T05:01:12.857082+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:12.857082+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7654,"log_dups_size":0,"ondisk_log_size":7654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41579680,"num_objects":5076,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7652,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2763}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7787","reported_seq":9963,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.908088+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:12.908088+0000","last_peered":"2024-09-11T05:01:12.908088+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:12.908088+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268384,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7785,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1056,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7832","reported_seq":7925,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.372834+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:14.372834+0000","last_peered":"2024-09-11T05:01:14.372834+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:14.372834+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40593728,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7829,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7615","reported_seq":7715,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.389206+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:15.389206+0000","last_peered":"2024-09-11T05:01:15.389206+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:15.389206+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244384,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7700","reported_seq":7789,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.494844+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:14.494844+0000","last_peered":"2024-09-11T05:01:14.494844+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:14.494844+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40570608,"num_objects":4953,"num_object_clones":0,"num_object_copies":19812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7455","reported_seq":12050,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:13.831312+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:13.831312+0000","last_peered":"2024-09-11T05:01:13.831312+0000","last_clean":"2024-09-11T05:01:13.831312+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:13.831312+0000","last_undegraded":"2024-09-11T05:01:13.831312+0000","last_fullsized":"2024-09-11T05:01:13.831312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40378368,"num_objects":4929,"num_object_clones":0,"num_object_copies":19716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1265,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7759","reported_seq":7850,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.256101+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:15.256101+0000","last_peered":"2024-09-11T05:01:15.256101+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:15.256101+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40663632,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2638}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7641","reported_seq":7720,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:14.464121+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:14.464121+0000","last_peered":"2024-09-11T05:01:14.464121+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:14.464121+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7641,"log_dups_size":0,"ondisk_log_size":7641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40555680,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7639,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2676}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7586","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.327570+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:15.327570+0000","last_peered":"2024-09-11T05:01:15.327570+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:15.327570+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39507104,"num_objects":4823,"num_object_clones":0,"num_object_copies":19292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2517}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7722","reported_seq":7800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:12.906239+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:12.906239+0000","last_peered":"2024-09-11T05:01:12.906239+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:12.906239+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41407648,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":648210056,"num_objects":79133,"num_object_clones":0,"num_object_copies":316532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116060,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":8577024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1237762048,"data_stored":1237762048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122276,"ondisk_log_size":122276,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746016,"kb_used_data":317256,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93625824,"statfs":{"total":96636764160,"available":95872843776,"internally_reserved":0,"allocated":324870144,"data_stored":324690175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740072,"kb_used_data":315992,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93631768,"statfs":{"total":96636764160,"available":95878930432,"internally_reserved":0,"allocated":323575808,"data_stored":323395839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,1,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540016,"kb_used_data":259868,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93831824,"statfs":{"total":96636764160,"available":96083787776,"internally_reserved":0,"allocated":266104832,"data_stored":265924863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":286846674},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742684,"kb_used_data":318156,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93629156,"statfs":{"total":96636764160,"available":95876255744,"internally_reserved":0,"allocated":325791744,"data_stored":325611775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,2,1,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325267456,"data_stored":325267456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265826304,"data_stored":265826304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322842624,"data_stored":322842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323825664,"data_stored":323825664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:18.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:18.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7627", 'reported_seq': 7690, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:14.368616+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:14.368616+0000', 'last_peered': '2024-09-11T05:01:14.368616+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:14.368616+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7627, 'log_dups_size': 0, 'ondisk_log_size': 7627, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40034304, 'num_objects': 4887, 'num_object_clones': 0, 'num_object_copies': 19548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7627, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2583}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:18.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:18.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7718", 'reported_seq': 7805, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:12.887789+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:12.887789+0000', 'last_peered': '2024-09-11T05:01:12.887789+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:12.887789+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7718, 'log_dups_size': 0, 'ondisk_log_size': 7718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41120928, 'num_objects': 5020, 'num_object_clones': 0, 'num_object_copies': 20080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2271, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7716, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2751}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:18.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:18.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7407", 'reported_seq': 7502, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:15.363551+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:15.363551+0000', 'last_peered': '2024-09-11T05:01:15.363551+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:15.363551+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7407, 'log_dups_size': 0, 'ondisk_log_size': 7407, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38935144, 'num_objects': 4754, 'num_object_clones': 0, 'num_object_copies': 19016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4754, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7406, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2541}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:18.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:18.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7523", 'reported_seq': 7601, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:12.833667+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:12.833667+0000', 'last_peered': '2024-09-11T05:01:12.833667+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:12.833667+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7523, 'log_dups_size': 0, 'ondisk_log_size': 7523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40080544, 'num_objects': 4893, 'num_object_clones': 0, 'num_object_copies': 19572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4893, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7521, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:18.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:18.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7541", 'reported_seq': 7622, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:14.398022+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:14.398022+0000', 'last_peered': '2024-09-11T05:01:14.398022+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:14.398022+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7541, 'log_dups_size': 0, 'ondisk_log_size': 7541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40875168, 'num_objects': 4990, 'num_object_clones': 0, 'num_object_copies': 19960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7539, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:18.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:18.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7709", 'reported_seq': 7791, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:14.427129+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:14.427129+0000', 'last_peered': '2024-09-11T05:01:14.427129+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:14.427129+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7709, 'log_dups_size': 0, 'ondisk_log_size': 7709, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40394752, 'num_objects': 4931, 'num_object_clones': 0, 'num_object_copies': 19724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2303, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7709, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2628}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:18.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:18.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7654", 'reported_seq': 7730, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:12.857082+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:12.857082+0000', 'last_peered': '2024-09-11T05:01:12.857082+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:12.857082+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7654, 'log_dups_size': 0, 'ondisk_log_size': 7654, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41579680, 'num_objects': 5076, 'num_object_clones': 0, 'num_object_copies': 20304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7652, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2763}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:18.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:18.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7787", 'reported_seq': 9963, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:12.908088+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:12.908088+0000', 'last_peered': '2024-09-11T05:01:12.908088+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:12.908088+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7787, 'log_dups_size': 0, 'ondisk_log_size': 7787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41268384, 'num_objects': 5038, 'num_object_clones': 0, 'num_object_copies': 20152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7785, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1056, 'num_bytes_recovered': 8577024, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3729}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:18.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:18.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7832", 'reported_seq': 7925, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:14.372834+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:14.372834+0000', 'last_peered': '2024-09-11T05:01:14.372834+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:14.372834+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7832, 'log_dups_size': 0, 'ondisk_log_size': 7832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40593728, 'num_objects': 4956, 'num_object_clones': 0, 'num_object_copies': 19824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2381, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7829, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2578}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:18.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:18.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7615", 'reported_seq': 7715, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:15.389206+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:15.389206+0000', 'last_peered': '2024-09-11T05:01:15.389206+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:15.389206+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7615, 'log_dups_size': 0, 'ondisk_log_size': 7615, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40244384, 'num_objects': 4913, 'num_object_clones': 0, 'num_object_copies': 19652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7613, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2605}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:18.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:18.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7700", 'reported_seq': 7789, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:14.494844+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:14.494844+0000', 'last_peered': '2024-09-11T05:01:14.494844+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:14.494844+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7700, 'log_dups_size': 0, 'ondisk_log_size': 7700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40570608, 'num_objects': 4953, 'num_object_clones': 0, 'num_object_copies': 19812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7697, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2644}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:18.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:18.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7759", 'reported_seq': 7850, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:15.256101+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:15.256101+0000', 'last_peered': '2024-09-11T05:01:15.256101+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:15.256101+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7759, 'log_dups_size': 0, 'ondisk_log_size': 7759, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40663632, 'num_objects': 4964, 'num_object_clones': 0, 'num_object_copies': 19856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7758, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2638}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:18.448 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:18.449 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7641", 'reported_seq': 7720, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:14.464121+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:14.464121+0000', 'last_peered': '2024-09-11T05:01:14.464121+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:14.464121+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7641, 'log_dups_size': 0, 'ondisk_log_size': 7641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40555680, 'num_objects': 4951, 'num_object_clones': 0, 'num_object_copies': 19804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7639, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:18.449 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:18.449 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7586", 'reported_seq': 7693, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:15.327570+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:15.327570+0000', 'last_peered': '2024-09-11T05:01:15.327570+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:15.327570+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7586, 'log_dups_size': 0, 'ondisk_log_size': 7586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39507104, 'num_objects': 4823, 'num_object_clones': 0, 'num_object_copies': 19292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7584, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2517}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:18.449 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:18.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7722", 'reported_seq': 7800, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:12.906239+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:12.906239+0000', 'last_peered': '2024-09-11T05:01:12.906239+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:12.906239+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7722, 'log_dups_size': 0, 'ondisk_log_size': 7722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41407648, 'num_objects': 5055, 'num_object_clones': 0, 'num_object_copies': 20220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7720, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:18.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:19.199 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:19.199 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33455/316532 objects degraded (10.569%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":406,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79133,"data_bytes":648210056,"bytes_used":2835238912,"bytes_avail":383711817728,"bytes_total":386547056640,"degraded_objects":33455,"degraded_total":316532,"degraded_ratio":0.10569231546889414,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":69601,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":835584,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:19.228 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:19.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:19.873 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:19.893 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-11T05:01:19.743092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":640730760,"num_objects":78220,"num_object_clones":0,"num_object_copies":312880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116973,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":8929280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123189,"ondisk_log_size":123189,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789464,"kb_used_data":1201408,"kb_used_omap":14,"kb_used_meta":1587953,"kb_avail":374697896,"statfs":{"total":386547056640,"available":383690645504,"internally_reserved":0,"allocated":1230241792,"data_stored":1229521916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1626064438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,6,5,8,13,1,0,0,0,0,0,2,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26648576,"num_objects":-3253,"num_object_clones":0,"num_object_copies":-13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3253,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005131"},"pg_stats":[{"pgid":"1.f","version":"33'7697","reported_seq":7760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.538725+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:19.538725+0000","last_peered":"2024-09-11T05:01:19.538725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:19.538725+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39460864,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7786","reported_seq":7877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.755505+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:17.755505+0000","last_peered":"2024-09-11T05:01:17.755505+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:17.755505+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7786,"log_dups_size":0,"ondisk_log_size":7786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40563872,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7784,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7407","reported_seq":7502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.363551+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:15.363551+0000","last_peered":"2024-09-11T05:01:15.363551+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:15.363551+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7407,"log_dups_size":0,"ondisk_log_size":7407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38935144,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7406,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2541}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7580","reported_seq":7660,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.781063+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:17.781063+0000","last_peered":"2024-09-11T05:01:17.781063+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:17.781063+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39613600,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7609","reported_seq":7690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.434037+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:19.434037+0000","last_peered":"2024-09-11T05:01:19.434037+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:19.434037+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40318112,"num_objects":4922,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7788","reported_seq":7870,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.505116+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:19.505116+0000","last_peered":"2024-09-11T05:01:19.505116+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:19.505116+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39747584,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7736","reported_seq":7814,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.827560+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:17.827560+0000","last_peered":"2024-09-11T05:01:17.827560+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:17.827560+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7736,"log_dups_size":0,"ondisk_log_size":7736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40907936,"num_objects":4994,"num_object_clones":0,"num_object_copies":19976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7734,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7880","reported_seq":10139,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.976119+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:17.976119+0000","last_peered":"2024-09-11T05:01:17.976119+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:17.976119+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7880,"log_dups_size":0,"ondisk_log_size":7880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40506528,"num_objects":4945,"num_object_clones":0,"num_object_copies":19780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1099,"num_bytes_recovered":8929280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3679}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7917","reported_seq":8010,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.471579+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:19.471579+0000","last_peered":"2024-09-11T05:01:19.471579+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:19.471579+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39897408,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7615","reported_seq":7715,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.389206+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:15.389206+0000","last_peered":"2024-09-11T05:01:15.389206+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:15.389206+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244384,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7777","reported_seq":7866,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.518467+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:19.518467+0000","last_peered":"2024-09-11T05:01:19.518467+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:19.518467+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39939824,"num_objects":4876,"num_object_clones":0,"num_object_copies":19504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2567}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7523","reported_seq":12118,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:18.826629+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:18.826629+0000","last_peered":"2024-09-11T05:01:18.826629+0000","last_clean":"2024-09-11T05:01:18.826629+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:18.826629+0000","last_undegraded":"2024-09-11T05:01:18.826629+0000","last_fullsized":"2024-09-11T05:01:18.826629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39821312,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1333,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7759","reported_seq":7850,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.256101+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:15.256101+0000","last_peered":"2024-09-11T05:01:15.256101+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:15.256101+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40663632,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7758,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2638}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7724","reported_seq":7803,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.557524+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:19.557524+0000","last_peered":"2024-09-11T05:01:19.557524+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:19.557524+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875744,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7586","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:15.327570+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:15.327570+0000","last_peered":"2024-09-11T05:01:15.327570+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:15.327570+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39507104,"num_objects":4823,"num_object_clones":0,"num_object_copies":19292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2517}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7805","reported_seq":7885,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.801180+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:17.801180+0000","last_peered":"2024-09-11T05:01:17.801180+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:17.801180+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7805,"log_dups_size":0,"ondisk_log_size":7805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40727712,"num_objects":4972,"num_object_clones":0,"num_object_copies":19888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7803,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":640730760,"num_objects":78220,"num_object_clones":0,"num_object_copies":312880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116973,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":8929280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1227644928,"data_stored":1227644928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123189,"ondisk_log_size":123189,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745328,"kb_used_data":312412,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93626512,"statfs":{"total":96636764160,"available":95873548288,"internally_reserved":0,"allocated":319909888,"data_stored":319729919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740072,"kb_used_data":315992,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93631768,"statfs":{"total":96636764160,"available":95878930432,"internally_reserved":0,"allocated":323575808,"data_stored":323395839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,1,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562020,"kb_used_data":259668,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93809820,"statfs":{"total":96636764160,"available":96061255680,"internally_reserved":0,"allocated":265900032,"data_stored":265720063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742044,"kb_used_data":313336,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93629796,"statfs":{"total":96636764160,"available":95876911104,"internally_reserved":0,"allocated":320856064,"data_stored":320676095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,1,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320331776,"data_stored":320331776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265613312,"data_stored":265613312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322842624,"data_stored":322842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318857216,"data_stored":318857216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:22.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:23.548 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-11T05:01:21.744029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":638305928,"num_objects":77924,"num_object_clones":0,"num_object_copies":311696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77924,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117269,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1269,"num_bytes_recovered":8929280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123485,"ondisk_log_size":123485,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788844,"kb_used_data":1196628,"kb_used_omap":14,"kb_used_meta":1592113,"kb_avail":374698516,"statfs":{"total":386547056640,"available":383691280384,"internally_reserved":0,"allocated":1225347072,"data_stored":1224627196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1630324278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,8,5,3,10,14,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21757952,"num_objects":-2656,"num_object_clones":0,"num_object_copies":-10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005264"},"pg_stats":[{"pgid":"1.f","version":"33'7697","reported_seq":7760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.538725+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:19.538725+0000","last_peered":"2024-09-11T05:01:19.538725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:19.538725+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39460864,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7786","reported_seq":7877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.755505+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:17.755505+0000","last_peered":"2024-09-11T05:01:17.755505+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:17.755505+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7786,"log_dups_size":0,"ondisk_log_size":7786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40563872,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7784,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7478","reported_seq":7573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.279114+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:20.279114+0000","last_peered":"2024-09-11T05:01:20.279114+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:20.279114+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38353512,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7477,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2470}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7580","reported_seq":7660,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.781063+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:17.781063+0000","last_peered":"2024-09-11T05:01:17.781063+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:17.781063+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39613600,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7609","reported_seq":7690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.434037+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:19.434037+0000","last_peered":"2024-09-11T05:01:19.434037+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:19.434037+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40318112,"num_objects":4922,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7788","reported_seq":7870,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.505116+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:19.505116+0000","last_peered":"2024-09-11T05:01:19.505116+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:19.505116+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39747584,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7736","reported_seq":7814,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.827560+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:17.827560+0000","last_peered":"2024-09-11T05:01:17.827560+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:17.827560+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7736,"log_dups_size":0,"ondisk_log_size":7736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40907936,"num_objects":4994,"num_object_clones":0,"num_object_copies":19976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7734,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7880","reported_seq":10139,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.976119+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:17.976119+0000","last_peered":"2024-09-11T05:01:17.976119+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:17.976119+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7880,"log_dups_size":0,"ondisk_log_size":7880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40506528,"num_objects":4945,"num_object_clones":0,"num_object_copies":19780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1099,"num_bytes_recovered":8929280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3679}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7917","reported_seq":8010,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.471579+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:19.471579+0000","last_peered":"2024-09-11T05:01:19.471579+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:19.471579+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39897408,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7695","reported_seq":7800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.451397+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:20.451397+0000","last_peered":"2024-09-11T05:01:20.451397+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:20.451397+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39589024,"num_objects":4833,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7777","reported_seq":7866,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.518467+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:19.518467+0000","last_peered":"2024-09-11T05:01:19.518467+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:19.518467+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39939824,"num_objects":4876,"num_object_clones":0,"num_object_copies":19504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2567}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7523","reported_seq":12118,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:18.826629+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:18.826629+0000","last_peered":"2024-09-11T05:01:18.826629+0000","last_clean":"2024-09-11T05:01:18.826629+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:18.826629+0000","last_undegraded":"2024-09-11T05:01:18.826629+0000","last_fullsized":"2024-09-11T05:01:18.826629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39821312,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1333,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7824","reported_seq":7915,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.276669+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:20.276669+0000","last_peered":"2024-09-11T05:01:20.276669+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:20.276669+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40131152,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7823,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2573}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7724","reported_seq":7803,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.557524+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:19.557524+0000","last_peered":"2024-09-11T05:01:19.557524+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:19.557524+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875744,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7666","reported_seq":7773,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.428299+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:20.428299+0000","last_peered":"2024-09-11T05:01:20.428299+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:20.428299+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851744,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7664,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2438}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7805","reported_seq":7885,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:17.801180+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:17.801180+0000","last_peered":"2024-09-11T05:01:17.801180+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:17.801180+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7805,"log_dups_size":0,"ondisk_log_size":7805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40727712,"num_objects":4972,"num_object_clones":0,"num_object_copies":19888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7803,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":638305928,"num_objects":77924,"num_object_clones":0,"num_object_copies":311696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77924,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117269,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1269,"num_bytes_recovered":8929280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1222766592,"data_stored":1222766592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123485,"ondisk_log_size":123485,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745328,"kb_used_data":312412,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93626512,"statfs":{"total":96636764160,"available":95873548288,"internally_reserved":0,"allocated":319909888,"data_stored":319729919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739452,"kb_used_data":311212,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93632388,"statfs":{"total":96636764160,"available":95879565312,"internally_reserved":0,"allocated":318681088,"data_stored":318501119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562020,"kb_used_data":259668,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93809820,"statfs":{"total":96636764160,"available":96061255680,"internally_reserved":0,"allocated":265900032,"data_stored":265720063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742044,"kb_used_data":313336,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93629796,"statfs":{"total":96636764160,"available":95876911104,"internally_reserved":0,"allocated":320856064,"data_stored":320676095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,1,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320331776,"data_stored":320331776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265613312,"data_stored":265613312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317964288,"data_stored":317964288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318857216,"data_stored":318857216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:23.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:24.203 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:24.233 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-11T05:01:23.744869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635184776,"num_objects":77543,"num_object_clones":0,"num_object_copies":310172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117650,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123866,"ondisk_log_size":123866,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788324,"kb_used_data":1191968,"kb_used_omap":14,"kb_used_meta":1596273,"kb_avail":374699036,"statfs":{"total":386547056640,"available":383691812864,"internally_reserved":0,"allocated":1220575232,"data_stored":1219855356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1634584118},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,2,5,12,8,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22519808,"num_objects":-2749,"num_object_clones":0,"num_object_copies":-10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005263"},"pg_stats":[{"pgid":"1.f","version":"33'7697","reported_seq":7760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.538725+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:19.538725+0000","last_peered":"2024-09-11T05:01:19.538725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:19.538725+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39460864,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7859","reported_seq":7950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:23.044223+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:23.044223+0000","last_peered":"2024-09-11T05:01:23.044223+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:23.044223+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965856,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7478","reported_seq":7573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.279114+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:20.279114+0000","last_peered":"2024-09-11T05:01:20.279114+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:20.279114+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38353512,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7477,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2470}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7660","reported_seq":7740,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.974012+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:22.974012+0000","last_peered":"2024-09-11T05:01:22.974012+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:22.974012+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7660,"log_dups_size":0,"ondisk_log_size":7660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958240,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7658,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2484}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7609","reported_seq":7690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.434037+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:19.434037+0000","last_peered":"2024-09-11T05:01:19.434037+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:19.434037+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40318112,"num_objects":4922,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7788","reported_seq":7870,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.505116+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:19.505116+0000","last_peered":"2024-09-11T05:01:19.505116+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:19.505116+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39747584,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7816","reported_seq":7894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.923212+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:22.923212+0000","last_peered":"2024-09-11T05:01:22.923212+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:22.923212+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7816,"log_dups_size":0,"ondisk_log_size":7816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40252576,"num_objects":4914,"num_object_clones":0,"num_object_copies":19656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7814,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7945","reported_seq":10309,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:23.021051+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:23.021051+0000","last_peered":"2024-09-11T05:01:23.021051+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:23.021051+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7945,"log_dups_size":0,"ondisk_log_size":7945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7917","reported_seq":8010,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.471579+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:19.471579+0000","last_peered":"2024-09-11T05:01:19.471579+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:19.471579+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39897408,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7695","reported_seq":7800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.451397+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:20.451397+0000","last_peered":"2024-09-11T05:01:20.451397+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:20.451397+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39589024,"num_objects":4833,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7777","reported_seq":7866,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.518467+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:19.518467+0000","last_peered":"2024-09-11T05:01:19.518467+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:19.518467+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39939824,"num_objects":4876,"num_object_clones":0,"num_object_copies":19504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2567}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7523","reported_seq":12118,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:18.826629+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:18.826629+0000","last_peered":"2024-09-11T05:01:18.826629+0000","last_clean":"2024-09-11T05:01:18.826629+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:18.826629+0000","last_undegraded":"2024-09-11T05:01:18.826629+0000","last_fullsized":"2024-09-11T05:01:18.826629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39821312,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1333,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7824","reported_seq":7915,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.276669+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:20.276669+0000","last_peered":"2024-09-11T05:01:20.276669+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:20.276669+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40131152,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7823,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2573}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7724","reported_seq":7803,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.557524+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:19.557524+0000","last_peered":"2024-09-11T05:01:19.557524+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:19.557524+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875744,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7666","reported_seq":7773,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.428299+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:20.428299+0000","last_peered":"2024-09-11T05:01:20.428299+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:20.428299+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851744,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7664,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2438}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7888","reported_seq":7970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.893169+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:22.893169+0000","last_peered":"2024-09-11T05:01:22.893169+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:22.893169+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40047776,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635184776,"num_objects":77543,"num_object_clones":0,"num_object_copies":310172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117650,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1217847296,"data_stored":1217847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123866,"ondisk_log_size":123866,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745328,"kb_used_data":312412,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93626512,"statfs":{"total":96636764160,"available":95873548288,"internally_reserved":0,"allocated":319909888,"data_stored":319729919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739452,"kb_used_data":311212,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93632388,"statfs":{"total":96636764160,"available":95879565312,"internally_reserved":0,"allocated":318681088,"data_stored":318501119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562020,"kb_used_data":259668,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93809820,"statfs":{"total":96636764160,"available":96061255680,"internally_reserved":0,"allocated":265900032,"data_stored":265720063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741524,"kb_used_data":308676,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93630316,"statfs":{"total":96636764160,"available":95877443584,"internally_reserved":0,"allocated":316084224,"data_stored":315904255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315412480,"data_stored":315412480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265613312,"data_stored":265613312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317964288,"data_stored":317964288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318857216,"data_stored":318857216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:24.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:24.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7697", 'reported_seq': 7760, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:19.538725+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:19.538725+0000', 'last_peered': '2024-09-11T05:01:19.538725+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:19.538725+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7697, 'log_dups_size': 0, 'ondisk_log_size': 7697, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39460864, 'num_objects': 4817, 'num_object_clones': 0, 'num_object_copies': 19268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7697, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2513}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:24.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:24.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7859", 'reported_seq': 7950, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:23.044223+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:23.044223+0000', 'last_peered': '2024-09-11T05:01:23.044223+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:23.044223+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7859, 'log_dups_size': 0, 'ondisk_log_size': 7859, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39965856, 'num_objects': 4879, 'num_object_clones': 0, 'num_object_copies': 19516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7857, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7478", 'reported_seq': 7573, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:20.279114+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:20.279114+0000', 'last_peered': '2024-09-11T05:01:20.279114+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:20.279114+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7478, 'log_dups_size': 0, 'ondisk_log_size': 7478, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38353512, 'num_objects': 4683, 'num_object_clones': 0, 'num_object_copies': 18732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4683, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7477, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2470}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7660", 'reported_seq': 7740, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:22.974012+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:22.974012+0000', 'last_peered': '2024-09-11T05:01:22.974012+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:22.974012+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7660, 'log_dups_size': 0, 'ondisk_log_size': 7660, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38958240, 'num_objects': 4756, 'num_object_clones': 0, 'num_object_copies': 19024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7658, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2484}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7609", 'reported_seq': 7690, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:19.434037+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:19.434037+0000', 'last_peered': '2024-09-11T05:01:19.434037+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:19.434037+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7609, 'log_dups_size': 0, 'ondisk_log_size': 7609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40318112, 'num_objects': 4922, 'num_object_clones': 0, 'num_object_copies': 19688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7607, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2683}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7788", 'reported_seq': 7870, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:19.505116+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:19.505116+0000', 'last_peered': '2024-09-11T05:01:19.505116+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:19.505116+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7788, 'log_dups_size': 0, 'ondisk_log_size': 7788, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39747584, 'num_objects': 4852, 'num_object_clones': 0, 'num_object_copies': 19408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2303, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7788, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2549}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:24.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7816", 'reported_seq': 7894, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:22.923212+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:22.923212+0000', 'last_peered': '2024-09-11T05:01:22.923212+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:22.923212+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7816, 'log_dups_size': 0, 'ondisk_log_size': 7816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40252576, 'num_objects': 4914, 'num_object_clones': 0, 'num_object_copies': 19656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7814, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2602}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:24.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:24.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7945", 'reported_seq': 10309, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:23.021051+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:23.021051+0000', 'last_peered': '2024-09-11T05:01:23.021051+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:23.021051+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7945, 'log_dups_size': 0, 'ondisk_log_size': 7945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39974048, 'num_objects': 4880, 'num_object_clones': 0, 'num_object_copies': 19520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7943, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1150, 'num_bytes_recovered': 9347072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3665}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:24.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:24.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7917", 'reported_seq': 8010, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:19.471579+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:19.471579+0000', 'last_peered': '2024-09-11T05:01:19.471579+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:19.471579+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7917, 'log_dups_size': 0, 'ondisk_log_size': 7917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39897408, 'num_objects': 4871, 'num_object_clones': 0, 'num_object_copies': 19484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2381, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7914, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2493}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:24.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:24.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7695", 'reported_seq': 7800, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:20.451397+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:20.451397+0000', 'last_peered': '2024-09-11T05:01:20.451397+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:20.451397+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7695, 'log_dups_size': 0, 'ondisk_log_size': 7695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39589024, 'num_objects': 4833, 'num_object_clones': 0, 'num_object_copies': 19332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7693, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2526}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7777", 'reported_seq': 7866, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:19.518467+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:19.518467+0000', 'last_peered': '2024-09-11T05:01:19.518467+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:19.518467+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7777, 'log_dups_size': 0, 'ondisk_log_size': 7777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39939824, 'num_objects': 4876, 'num_object_clones': 0, 'num_object_copies': 19504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7774, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2567}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7824", 'reported_seq': 7915, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:20.276669+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:20.276669+0000', 'last_peered': '2024-09-11T05:01:20.276669+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:20.276669+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7824, 'log_dups_size': 0, 'ondisk_log_size': 7824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40131152, 'num_objects': 4899, 'num_object_clones': 0, 'num_object_copies': 19596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7823, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2573}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7724", 'reported_seq': 7803, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:19.557524+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:19.557524+0000', 'last_peered': '2024-09-11T05:01:19.557524+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:19.557524+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7724, 'log_dups_size': 0, 'ondisk_log_size': 7724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39875744, 'num_objects': 4868, 'num_object_clones': 0, 'num_object_copies': 19472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7722, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7666", 'reported_seq': 7773, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:20.428299+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:20.428299+0000', 'last_peered': '2024-09-11T05:01:20.428299+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:20.428299+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7666, 'log_dups_size': 0, 'ondisk_log_size': 7666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38851744, 'num_objects': 4743, 'num_object_clones': 0, 'num_object_copies': 18972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7664, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2438}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:24.238 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7888", 'reported_seq': 7970, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:22.893169+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:22.893169+0000', 'last_peered': '2024-09-11T05:01:22.893169+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:22.893169+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7888, 'log_dups_size': 0, 'ondisk_log_size': 7888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40047776, 'num_objects': 4889, 'num_object_clones': 0, 'num_object_copies': 19556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7886, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2585}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:24.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:25.007 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:25.008 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33353/310172 objects degraded (10.753%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":412,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77543,"data_bytes":635184776,"bytes_used":2855243776,"bytes_avail":383691812864,"bytes_total":386547056640,"degraded_objects":33353,"degraded_total":310172,"degraded_ratio":0.10753066040777375,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":102355,"recovering_keys_per_sec":0,"num_objects_recovered":168,"num_bytes_recovered":1228800,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":228},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:25.036 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:25.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:25.705 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:25.705 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:25.734 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-11T05:01:23.744869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635184776,"num_objects":77543,"num_object_clones":0,"num_object_copies":310172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117650,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123866,"ondisk_log_size":123866,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788324,"kb_used_data":1191968,"kb_used_omap":14,"kb_used_meta":1596273,"kb_avail":374699036,"statfs":{"total":386547056640,"available":383691812864,"internally_reserved":0,"allocated":1220575232,"data_stored":1219855356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1634584118},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,2,5,12,8,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22519808,"num_objects":-2749,"num_object_clones":0,"num_object_copies":-10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005263"},"pg_stats":[{"pgid":"1.f","version":"33'7697","reported_seq":7760,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.538725+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:19.538725+0000","last_peered":"2024-09-11T05:01:19.538725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:19.538725+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39460864,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7859","reported_seq":7950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:23.044223+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:23.044223+0000","last_peered":"2024-09-11T05:01:23.044223+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:23.044223+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965856,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7478","reported_seq":7573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.279114+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:20.279114+0000","last_peered":"2024-09-11T05:01:20.279114+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:20.279114+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38353512,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7477,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2470}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7660","reported_seq":7740,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.974012+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:22.974012+0000","last_peered":"2024-09-11T05:01:22.974012+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:22.974012+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7660,"log_dups_size":0,"ondisk_log_size":7660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958240,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7658,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2484}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7609","reported_seq":7690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.434037+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:19.434037+0000","last_peered":"2024-09-11T05:01:19.434037+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:19.434037+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40318112,"num_objects":4922,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7788","reported_seq":7870,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.505116+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:19.505116+0000","last_peered":"2024-09-11T05:01:19.505116+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:19.505116+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39747584,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7816","reported_seq":7894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.923212+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:22.923212+0000","last_peered":"2024-09-11T05:01:22.923212+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:22.923212+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7816,"log_dups_size":0,"ondisk_log_size":7816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40252576,"num_objects":4914,"num_object_clones":0,"num_object_copies":19656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7814,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7945","reported_seq":10309,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:23.021051+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:23.021051+0000","last_peered":"2024-09-11T05:01:23.021051+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:23.021051+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7945,"log_dups_size":0,"ondisk_log_size":7945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7917","reported_seq":8010,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.471579+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:19.471579+0000","last_peered":"2024-09-11T05:01:19.471579+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:19.471579+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39897408,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7695","reported_seq":7800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.451397+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:20.451397+0000","last_peered":"2024-09-11T05:01:20.451397+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:20.451397+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39589024,"num_objects":4833,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7777","reported_seq":7866,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.518467+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:19.518467+0000","last_peered":"2024-09-11T05:01:19.518467+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:19.518467+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39939824,"num_objects":4876,"num_object_clones":0,"num_object_copies":19504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2567}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7523","reported_seq":12118,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:18.826629+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:18.826629+0000","last_peered":"2024-09-11T05:01:18.826629+0000","last_clean":"2024-09-11T05:01:18.826629+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:18.826629+0000","last_undegraded":"2024-09-11T05:01:18.826629+0000","last_fullsized":"2024-09-11T05:01:18.826629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39821312,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1333,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7824","reported_seq":7915,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.276669+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:20.276669+0000","last_peered":"2024-09-11T05:01:20.276669+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:20.276669+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40131152,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7823,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2573}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7724","reported_seq":7803,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:19.557524+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:19.557524+0000","last_peered":"2024-09-11T05:01:19.557524+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:19.557524+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875744,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7666","reported_seq":7773,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:20.428299+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:20.428299+0000","last_peered":"2024-09-11T05:01:20.428299+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:20.428299+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851744,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7664,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2438}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7888","reported_seq":7970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.893169+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:22.893169+0000","last_peered":"2024-09-11T05:01:22.893169+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:22.893169+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40047776,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635184776,"num_objects":77543,"num_object_clones":0,"num_object_copies":310172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117650,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1217847296,"data_stored":1217847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123866,"ondisk_log_size":123866,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745328,"kb_used_data":312412,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93626512,"statfs":{"total":96636764160,"available":95873548288,"internally_reserved":0,"allocated":319909888,"data_stored":319729919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2,1,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739452,"kb_used_data":311212,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93632388,"statfs":{"total":96636764160,"available":95879565312,"internally_reserved":0,"allocated":318681088,"data_stored":318501119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562020,"kb_used_data":259668,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93809820,"statfs":{"total":96636764160,"available":96061255680,"internally_reserved":0,"allocated":265900032,"data_stored":265720063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741524,"kb_used_data":308676,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93630316,"statfs":{"total":96636764160,"available":95877443584,"internally_reserved":0,"allocated":316084224,"data_stored":315904255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315412480,"data_stored":315412480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265613312,"data_stored":265613312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317964288,"data_stored":317964288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318857216,"data_stored":318857216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:28.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:29.389 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:29.389 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:29.427 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-11T05:01:27.746469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":628942472,"num_objects":76781,"num_object_clones":0,"num_object_copies":307124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118412,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1323,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124628,"ondisk_log_size":124628,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799768,"kb_used_data":1182612,"kb_used_omap":14,"kb_used_meta":1617073,"kb_avail":374687592,"statfs":{"total":386547056640,"available":383680094208,"internally_reserved":0,"allocated":1210994688,"data_stored":1210274812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1655883318},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,16,10,6,2,0,0,1,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19267584,"num_objects":-2352,"num_object_clones":0,"num_object_copies":-9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005123"},"pg_stats":[{"pgid":"1.f","version":"33'7766","reported_seq":7829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:24.552751+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:24.552751+0000","last_peered":"2024-09-11T05:01:24.552751+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:24.552751+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7766,"log_dups_size":0,"ondisk_log_size":7766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38895616,"num_objects":4748,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7859","reported_seq":7950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:23.044223+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:23.044223+0000","last_peered":"2024-09-11T05:01:23.044223+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:23.044223+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965856,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7560","reported_seq":7655,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.409894+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:25.409894+0000","last_peered":"2024-09-11T05:01:25.409894+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:25.409894+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37681768,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7559,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2388}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7660","reported_seq":7740,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.974012+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:22.974012+0000","last_peered":"2024-09-11T05:01:22.974012+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:22.974012+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7660,"log_dups_size":0,"ondisk_log_size":7660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958240,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7658,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2484}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7673","reported_seq":7754,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:24.498810+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:24.498810+0000","last_peered":"2024-09-11T05:01:24.498810+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:24.498810+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39793824,"num_objects":4858,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7855","reported_seq":7937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:24.527836+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:24.527836+0000","last_peered":"2024-09-11T05:01:24.527836+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:24.527836+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7855,"log_dups_size":0,"ondisk_log_size":7855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39198720,"num_objects":4785,"num_object_clones":0,"num_object_copies":19140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7816","reported_seq":7894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.923212+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:22.923212+0000","last_peered":"2024-09-11T05:01:22.923212+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:22.923212+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7816,"log_dups_size":0,"ondisk_log_size":7816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40252576,"num_objects":4914,"num_object_clones":0,"num_object_copies":19656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7814,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7945","reported_seq":10309,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:23.021051+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:23.021051+0000","last_peered":"2024-09-11T05:01:23.021051+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:23.021051+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7945,"log_dups_size":0,"ondisk_log_size":7945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974048,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8002","reported_seq":8099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:24.480409+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:24.480409+0000","last_peered":"2024-09-11T05:01:24.480409+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:24.480409+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39201088,"num_objects":4786,"num_object_clones":0,"num_object_copies":19144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7999,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2410}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7764","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.440555+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:25.440555+0000","last_peered":"2024-09-11T05:01:25.440555+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:25.440555+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7846","reported_seq":7935,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:24.169196+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:24.169196+0000","last_peered":"2024-09-11T05:01:24.169196+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:24.169196+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7846,"log_dups_size":0,"ondisk_log_size":7846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39374576,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7843,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7579","reported_seq":12173,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:23.926048+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:23.926048+0000","last_peered":"2024-09-11T05:01:23.926048+0000","last_clean":"2024-09-11T05:01:23.926048+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:23.926048+0000","last_undegraded":"2024-09-11T05:01:23.926048+0000","last_fullsized":"2024-09-11T05:01:23.926048+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7579,"log_dups_size":0,"ondisk_log_size":7579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39362560,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1389,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7891","reported_seq":7982,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.312347+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:25.312347+0000","last_peered":"2024-09-11T05:01:25.312347+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:25.312347+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7891,"log_dups_size":0,"ondisk_log_size":7891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39582288,"num_objects":4832,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7890,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2506}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7793","reported_seq":7872,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:24.361059+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:24.361059+0000","last_peered":"2024-09-11T05:01:24.361059+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:24.361059+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7793,"log_dups_size":0,"ondisk_log_size":7793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39310496,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7791,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2524}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7731","reported_seq":7840,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.049926+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:25.049926+0000","last_peered":"2024-09-11T05:01:25.049926+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:25.049926+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7731,"log_dups_size":0,"ondisk_log_size":7731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38319264,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7729,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7888","reported_seq":7970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:22.893169+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:22.893169+0000","last_peered":"2024-09-11T05:01:22.893169+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:22.893169+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40047776,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":628942472,"num_objects":76781,"num_object_clones":0,"num_object_copies":307124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118412,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1323,"num_bytes_recovered":9347072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1208320000,"data_stored":1208320000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124628,"ondisk_log_size":124628,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744828,"kb_used_data":307752,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93627012,"statfs":{"total":96636764160,"available":95874060288,"internally_reserved":0,"allocated":315138048,"data_stored":314958079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,11,1,2,2,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743248,"kb_used_data":306688,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93628592,"statfs":{"total":96636764160,"available":95875678208,"internally_reserved":0,"allocated":314048512,"data_stored":313868543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570168,"kb_used_data":259496,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93801672,"statfs":{"total":96636764160,"available":96052912128,"internally_reserved":0,"allocated":265723904,"data_stored":265543935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318107346},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741524,"kb_used_data":308676,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93630316,"statfs":{"total":96636764160,"available":95877443584,"internally_reserved":0,"allocated":316084224,"data_stored":315904255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315412480,"data_stored":315412480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265408512,"data_stored":265408512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313376768,"data_stored":313376768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314122240,"data_stored":314122240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:29.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:30.037 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:30.058 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-11T05:01:29.747387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622093960,"num_objects":75945,"num_object_clones":0,"num_object_copies":303780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119248,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":9822208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125464,"ondisk_log_size":125464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773312,"kb_used_data":1172784,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374714048,"statfs":{"total":386547056640,"available":383707185152,"internally_reserved":0,"allocated":1200930816,"data_stored":1200210940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1638843958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,16,11,7,5,2,2,0,1,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26116096,"num_objects":-3188,"num_object_clones":0,"num_object_copies":-12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005147"},"pg_stats":[{"pgid":"1.f","version":"33'7845","reported_seq":7908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.544816+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:29.544816+0000","last_peered":"2024-09-11T05:01:29.544816+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:29.544816+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7845,"log_dups_size":0,"ondisk_log_size":7845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38248448,"num_objects":4669,"num_object_clones":0,"num_object_copies":18676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7845,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7928","reported_seq":8023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:27.859900+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:27.859900+0000","last_peered":"2024-09-11T05:01:27.859900+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:27.859900+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7928,"log_dups_size":0,"ondisk_log_size":7928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39400608,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7560","reported_seq":7655,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.409894+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:25.409894+0000","last_peered":"2024-09-11T05:01:25.409894+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:25.409894+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37681768,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7559,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2388}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7724","reported_seq":7807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:28.035754+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:28.035754+0000","last_peered":"2024-09-11T05:01:28.035754+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:28.035754+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38433952,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7734","reported_seq":7815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.450863+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:29.450863+0000","last_peered":"2024-09-11T05:01:29.450863+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:29.450863+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39294112,"num_objects":4797,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7922","reported_seq":8004,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.352995+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:29.352995+0000","last_peered":"2024-09-11T05:01:29.352995+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:29.352995+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7922,"log_dups_size":0,"ondisk_log_size":7922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38649856,"num_objects":4718,"num_object_clones":0,"num_object_copies":18872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7889","reported_seq":7967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:27.960522+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:27.960522+0000","last_peered":"2024-09-11T05:01:27.960522+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:27.960522+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39654560,"num_objects":4841,"num_object_clones":0,"num_object_copies":19364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8027","reported_seq":10507,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:28.056947+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:28.056947+0000","last_peered":"2024-09-11T05:01:28.056947+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:28.056947+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8027,"log_dups_size":0,"ondisk_log_size":8027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302304,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8025,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":9822208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8062","reported_seq":8159,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.621154+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:29.621154+0000","last_peered":"2024-09-11T05:01:29.621154+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:29.621154+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8062,"log_dups_size":0,"ondisk_log_size":8062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38709568,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7764","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.440555+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:25.440555+0000","last_peered":"2024-09-11T05:01:25.440555+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:25.440555+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7911","reported_seq":8000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.571597+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:29.571597+0000","last_peered":"2024-09-11T05:01:29.571597+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:29.571597+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842096,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2433}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7652","reported_seq":12247,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:28.809042+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:28.809042+0000","last_peered":"2024-09-11T05:01:28.809042+0000","last_clean":"2024-09-11T05:01:28.809042+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:28.809042+0000","last_undegraded":"2024-09-11T05:01:28.809042+0000","last_fullsized":"2024-09-11T05:01:28.809042+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38764544,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1462,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7891","reported_seq":7982,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.312347+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:25.312347+0000","last_peered":"2024-09-11T05:01:25.312347+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:25.312347+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7891,"log_dups_size":0,"ondisk_log_size":7891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39582288,"num_objects":4832,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7890,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2506}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7870","reported_seq":7948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.566287+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:29.566287+0000","last_peered":"2024-09-11T05:01:29.566287+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:29.566287+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7870,"log_dups_size":0,"ondisk_log_size":7870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38679712,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7868,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7731","reported_seq":7840,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.049926+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:25.049926+0000","last_peered":"2024-09-11T05:01:25.049926+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:25.049926+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7731,"log_dups_size":0,"ondisk_log_size":7731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38319264,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7729,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7954","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:28.013260+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:28.013260+0000","last_peered":"2024-09-11T05:01:28.013260+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:28.013260+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7954,"log_dups_size":0,"ondisk_log_size":7954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39507104,"num_objects":4823,"num_object_clones":0,"num_object_copies":19292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7952,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2523}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622093960,"num_objects":75945,"num_object_clones":0,"num_object_copies":303780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119248,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":9822208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198977024,"data_stored":1198977024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125464,"ondisk_log_size":125464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731332,"kb_used_data":302564,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93640508,"statfs":{"total":96636764160,"available":95887880192,"internally_reserved":0,"allocated":309825536,"data_stored":309645567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,6,0,2,2,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743248,"kb_used_data":306688,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93628592,"statfs":{"total":96636764160,"available":95875678208,"internally_reserved":0,"allocated":314048512,"data_stored":313868543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557408,"kb_used_data":259216,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93814432,"statfs":{"total":96636764160,"available":96065978368,"internally_reserved":0,"allocated":265437184,"data_stored":265257215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":305327826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741324,"kb_used_data":304316,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93630516,"statfs":{"total":96636764160,"available":95877648384,"internally_reserved":0,"allocated":311619584,"data_stored":311439615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,3,3,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310943744,"data_stored":310943744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265138176,"data_stored":265138176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313376768,"data_stored":313376768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":309518336,"data_stored":309518336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:30.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7845", 'reported_seq': 7908, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:29.544816+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:29.544816+0000', 'last_peered': '2024-09-11T05:01:29.544816+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:29.544816+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7845, 'log_dups_size': 0, 'ondisk_log_size': 7845, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38248448, 'num_objects': 4669, 'num_object_clones': 0, 'num_object_copies': 18676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7845, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2365}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7928", 'reported_seq': 8023, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:27.859900+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:27.859900+0000', 'last_peered': '2024-09-11T05:01:27.859900+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:27.859900+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7928, 'log_dups_size': 0, 'ondisk_log_size': 7928, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39400608, 'num_objects': 4810, 'num_object_clones': 0, 'num_object_copies': 19240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2267, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7926, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7560", 'reported_seq': 7655, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:25.409894+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:25.409894+0000', 'last_peered': '2024-09-11T05:01:25.409894+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:25.409894+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7560, 'log_dups_size': 0, 'ondisk_log_size': 7560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37681768, 'num_objects': 4601, 'num_object_clones': 0, 'num_object_copies': 18404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4601, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7559, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2388}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7724", 'reported_seq': 7807, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:28.035754+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:28.035754+0000', 'last_peered': '2024-09-11T05:01:28.035754+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:28.035754+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7724, 'log_dups_size': 0, 'ondisk_log_size': 7724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38433952, 'num_objects': 4692, 'num_object_clones': 0, 'num_object_copies': 18768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7722, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:30.060 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7734", 'reported_seq': 7815, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:29.450863+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:29.450863+0000', 'last_peered': '2024-09-11T05:01:29.450863+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:29.450863+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7734, 'log_dups_size': 0, 'ondisk_log_size': 7734, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39294112, 'num_objects': 4797, 'num_object_clones': 0, 'num_object_copies': 19188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7732, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2558}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7922", 'reported_seq': 8004, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:29.352995+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:29.352995+0000', 'last_peered': '2024-09-11T05:01:29.352995+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:29.352995+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7922, 'log_dups_size': 0, 'ondisk_log_size': 7922, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38649856, 'num_objects': 4718, 'num_object_clones': 0, 'num_object_copies': 18872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2303, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7922, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2415}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7889", 'reported_seq': 7967, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:27.960522+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:27.960522+0000', 'last_peered': '2024-09-11T05:01:27.960522+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:27.960522+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7889, 'log_dups_size': 0, 'ondisk_log_size': 7889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39654560, 'num_objects': 4841, 'num_object_clones': 0, 'num_object_copies': 19364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7887, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2529}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8027", 'reported_seq': 10507, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:28.056947+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:28.056947+0000', 'last_peered': '2024-09-11T05:01:28.056947+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:28.056947+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8027, 'log_dups_size': 0, 'ondisk_log_size': 8027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39302304, 'num_objects': 4798, 'num_object_clones': 0, 'num_object_copies': 19192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1157, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8025, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1208, 'num_bytes_recovered': 9822208, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8062", 'reported_seq': 8159, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:29.621154+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:29.621154+0000', 'last_peered': '2024-09-11T05:01:29.621154+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:29.621154+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8062, 'log_dups_size': 0, 'ondisk_log_size': 8062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38709568, 'num_objects': 4726, 'num_object_clones': 0, 'num_object_copies': 18904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2379, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8059, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2350}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7764", 'reported_seq': 7871, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:25.440555+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:25.440555+0000', 'last_peered': '2024-09-11T05:01:25.440555+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:25.440555+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7764, 'log_dups_size': 0, 'ondisk_log_size': 7764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39023776, 'num_objects': 4764, 'num_object_clones': 0, 'num_object_copies': 19056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7762, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2458}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:30.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7911", 'reported_seq': 8000, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:29.571597+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:29.571597+0000', 'last_peered': '2024-09-11T05:01:29.571597+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:29.571597+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7911, 'log_dups_size': 0, 'ondisk_log_size': 7911, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38842096, 'num_objects': 4742, 'num_object_clones': 0, 'num_object_copies': 18968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7908, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2433}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7891", 'reported_seq': 7982, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:25.312347+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:25.312347+0000', 'last_peered': '2024-09-11T05:01:25.312347+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:25.312347+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7891, 'log_dups_size': 0, 'ondisk_log_size': 7891, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39582288, 'num_objects': 4832, 'num_object_clones': 0, 'num_object_copies': 19328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7890, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2506}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7870", 'reported_seq': 7948, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:29.566287+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:29.566287+0000', 'last_peered': '2024-09-11T05:01:29.566287+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:29.566287+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7870, 'log_dups_size': 0, 'ondisk_log_size': 7870, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38679712, 'num_objects': 4722, 'num_object_clones': 0, 'num_object_copies': 18888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7868, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2447}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7731", 'reported_seq': 7840, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:25.049926+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:25.049926+0000', 'last_peered': '2024-09-11T05:01:25.049926+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:25.049926+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7731, 'log_dups_size': 0, 'ondisk_log_size': 7731, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38319264, 'num_objects': 4678, 'num_object_clones': 0, 'num_object_copies': 18712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7729, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2373}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:30.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7954", 'reported_seq': 8044, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:28.013260+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:28.013260+0000', 'last_peered': '2024-09-11T05:01:28.013260+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:28.013260+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7954, 'log_dups_size': 0, 'ondisk_log_size': 7954, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39507104, 'num_objects': 4823, 'num_object_clones': 0, 'num_object_copies': 19292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2302, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7952, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:30.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:30.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:30.835 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33285/303780 objects degraded (10.957%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":417,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":75945,"data_bytes":622093960,"bytes_used":2839871488,"bytes_avail":383707185152,"bytes_total":386547056640,"degraded_objects":33285,"degraded_total":303780,"degraded_ratio":0.10956942524195142,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":103720,"recovering_keys_per_sec":0,"num_objects_recovered":170,"num_bytes_recovered":1245184,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":265},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:30.865 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:30.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:31.540 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:31.562 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-11T05:01:29.747387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622093960,"num_objects":75945,"num_object_clones":0,"num_object_copies":303780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119248,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":9822208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125464,"ondisk_log_size":125464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773312,"kb_used_data":1172784,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374714048,"statfs":{"total":386547056640,"available":383707185152,"internally_reserved":0,"allocated":1200930816,"data_stored":1200210940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1638843958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,16,11,7,5,2,2,0,1,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26116096,"num_objects":-3188,"num_object_clones":0,"num_object_copies":-12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005147"},"pg_stats":[{"pgid":"1.f","version":"33'7845","reported_seq":7908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.544816+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:29.544816+0000","last_peered":"2024-09-11T05:01:29.544816+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:29.544816+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7845,"log_dups_size":0,"ondisk_log_size":7845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38248448,"num_objects":4669,"num_object_clones":0,"num_object_copies":18676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7845,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7928","reported_seq":8023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:27.859900+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:27.859900+0000","last_peered":"2024-09-11T05:01:27.859900+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:27.859900+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7928,"log_dups_size":0,"ondisk_log_size":7928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39400608,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7560","reported_seq":7655,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.409894+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:25.409894+0000","last_peered":"2024-09-11T05:01:25.409894+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:25.409894+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37681768,"num_objects":4601,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7559,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2388}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7724","reported_seq":7807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:28.035754+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:28.035754+0000","last_peered":"2024-09-11T05:01:28.035754+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:28.035754+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38433952,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7734","reported_seq":7815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.450863+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:29.450863+0000","last_peered":"2024-09-11T05:01:29.450863+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:29.450863+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39294112,"num_objects":4797,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7922","reported_seq":8004,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.352995+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:29.352995+0000","last_peered":"2024-09-11T05:01:29.352995+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:29.352995+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7922,"log_dups_size":0,"ondisk_log_size":7922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38649856,"num_objects":4718,"num_object_clones":0,"num_object_copies":18872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7889","reported_seq":7967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:27.960522+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:27.960522+0000","last_peered":"2024-09-11T05:01:27.960522+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:27.960522+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39654560,"num_objects":4841,"num_object_clones":0,"num_object_copies":19364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8027","reported_seq":10507,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:28.056947+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:28.056947+0000","last_peered":"2024-09-11T05:01:28.056947+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:28.056947+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8027,"log_dups_size":0,"ondisk_log_size":8027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302304,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8025,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":9822208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8062","reported_seq":8159,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.621154+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:29.621154+0000","last_peered":"2024-09-11T05:01:29.621154+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:29.621154+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8062,"log_dups_size":0,"ondisk_log_size":8062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38709568,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7764","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.440555+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:25.440555+0000","last_peered":"2024-09-11T05:01:25.440555+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:25.440555+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7911","reported_seq":8000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.571597+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:29.571597+0000","last_peered":"2024-09-11T05:01:29.571597+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:29.571597+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842096,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2433}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7652","reported_seq":12247,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:28.809042+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:28.809042+0000","last_peered":"2024-09-11T05:01:28.809042+0000","last_clean":"2024-09-11T05:01:28.809042+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:28.809042+0000","last_undegraded":"2024-09-11T05:01:28.809042+0000","last_fullsized":"2024-09-11T05:01:28.809042+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38764544,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1462,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7891","reported_seq":7982,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.312347+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:25.312347+0000","last_peered":"2024-09-11T05:01:25.312347+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:25.312347+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7891,"log_dups_size":0,"ondisk_log_size":7891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39582288,"num_objects":4832,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7890,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2506}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7870","reported_seq":7948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.566287+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:29.566287+0000","last_peered":"2024-09-11T05:01:29.566287+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:29.566287+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7870,"log_dups_size":0,"ondisk_log_size":7870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38679712,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7868,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7731","reported_seq":7840,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:25.049926+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:25.049926+0000","last_peered":"2024-09-11T05:01:25.049926+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:25.049926+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7731,"log_dups_size":0,"ondisk_log_size":7731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38319264,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7729,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2373}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7954","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:28.013260+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:28.013260+0000","last_peered":"2024-09-11T05:01:28.013260+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:28.013260+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7954,"log_dups_size":0,"ondisk_log_size":7954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39507104,"num_objects":4823,"num_object_clones":0,"num_object_copies":19292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7952,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2523}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622093960,"num_objects":75945,"num_object_clones":0,"num_object_copies":303780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119248,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":9822208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198977024,"data_stored":1198977024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125464,"ondisk_log_size":125464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731332,"kb_used_data":302564,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93640508,"statfs":{"total":96636764160,"available":95887880192,"internally_reserved":0,"allocated":309825536,"data_stored":309645567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,6,0,2,2,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743248,"kb_used_data":306688,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93628592,"statfs":{"total":96636764160,"available":95875678208,"internally_reserved":0,"allocated":314048512,"data_stored":313868543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557408,"kb_used_data":259216,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93814432,"statfs":{"total":96636764160,"available":96065978368,"internally_reserved":0,"allocated":265437184,"data_stored":265257215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":305327826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741324,"kb_used_data":304316,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93630516,"statfs":{"total":96636764160,"available":95877648384,"internally_reserved":0,"allocated":311619584,"data_stored":311439615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,3,3,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310943744,"data_stored":310943744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265138176,"data_stored":265138176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313376768,"data_stored":313376768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":309518336,"data_stored":309518336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:34.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:35.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:35.215 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:35.235 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-11T05:01:33.748711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617170568,"num_objects":75344,"num_object_clones":0,"num_object_copies":301376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119849,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126065,"ondisk_log_size":126065,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743724,"kb_used_data":1163976,"kb_used_omap":14,"kb_used_meta":1579633,"kb_avail":374743636,"statfs":{"total":386547056640,"available":383737483264,"internally_reserved":0,"allocated":1191911424,"data_stored":1191191548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1617544758},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,13,7,6,6,2,0,0,0,0,0,3,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21135360,"num_objects":-2580,"num_object_clones":0,"num_object_copies":-10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2580,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1409024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004682"},"pg_stats":[{"pgid":"1.f","version":"33'7845","reported_seq":7908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.544816+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:29.544816+0000","last_peered":"2024-09-11T05:01:29.544816+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:29.544816+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7845,"log_dups_size":0,"ondisk_log_size":7845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38248448,"num_objects":4669,"num_object_clones":0,"num_object_copies":18676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7845,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8000","reported_seq":8095,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.989868+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:32.989868+0000","last_peered":"2024-09-11T05:01:32.989868+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:32.989868+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810784,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2473}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7617","reported_seq":7712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:30.408480+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:30.408480+0000","last_peered":"2024-09-11T05:01:30.408480+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:30.408480+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7617,"log_dups_size":0,"ondisk_log_size":7617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37214824,"num_objects":4544,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7616,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7788","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.934108+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:32.934108+0000","last_peered":"2024-09-11T05:01:32.934108+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:32.934108+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37909664,"num_objects":4628,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7786,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7734","reported_seq":7815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.450863+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:29.450863+0000","last_peered":"2024-09-11T05:01:29.450863+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:29.450863+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39294112,"num_objects":4797,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7922","reported_seq":8004,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.352995+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:29.352995+0000","last_peered":"2024-09-11T05:01:29.352995+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:29.352995+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7922,"log_dups_size":0,"ondisk_log_size":7922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38649856,"num_objects":4718,"num_object_clones":0,"num_object_copies":18872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7956","reported_seq":8034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.985948+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:32.985948+0000","last_peered":"2024-09-11T05:01:32.985948+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:32.985948+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7956,"log_dups_size":0,"ondisk_log_size":7956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39105696,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2462}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8092","reported_seq":10698,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:33.067033+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:33.067033+0000","last_peered":"2024-09-11T05:01:33.067033+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:33.067033+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38769824,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1271,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8062","reported_seq":8159,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.621154+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:29.621154+0000","last_peered":"2024-09-11T05:01:29.621154+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:29.621154+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8062,"log_dups_size":0,"ondisk_log_size":8062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38709568,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7848","reported_seq":7955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:30.491824+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:30.491824+0000","last_peered":"2024-09-11T05:01:30.491824+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:30.491824+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7848,"log_dups_size":0,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38335648,"num_objects":4680,"num_object_clones":0,"num_object_copies":18720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2374}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7911","reported_seq":8000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.571597+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:29.571597+0000","last_peered":"2024-09-11T05:01:29.571597+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:29.571597+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842096,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2433}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7652","reported_seq":12247,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:28.809042+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:28.809042+0000","last_peered":"2024-09-11T05:01:28.809042+0000","last_clean":"2024-09-11T05:01:28.809042+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:28.809042+0000","last_undegraded":"2024-09-11T05:01:28.809042+0000","last_fullsized":"2024-09-11T05:01:28.809042+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38764544,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1462,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7964","reported_seq":8055,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:30.354484+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:30.354484+0000","last_peered":"2024-09-11T05:01:30.354484+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:30.354484+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7964,"log_dups_size":0,"ondisk_log_size":7964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38984272,"num_objects":4759,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7963,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2433}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7870","reported_seq":7948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:29.566287+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:29.566287+0000","last_peered":"2024-09-11T05:01:29.566287+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:29.566287+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7870,"log_dups_size":0,"ondisk_log_size":7870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38679712,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7868,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2447}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7791","reported_seq":7900,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:30.362157+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:30.362157+0000","last_peered":"2024-09-11T05:01:30.362157+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:30.362157+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7791,"log_dups_size":0,"ondisk_log_size":7791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37827744,"num_objects":4618,"num_object_clones":0,"num_object_copies":18472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7789,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2313}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8013","reported_seq":8103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:33.012161+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:33.012161+0000","last_peered":"2024-09-11T05:01:33.012161+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:33.012161+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":617170568,"num_objects":75344,"num_object_clones":0,"num_object_copies":301376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119849,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1189986304,"data_stored":1189986304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126065,"ondisk_log_size":126065,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731332,"kb_used_data":302564,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93640508,"statfs":{"total":96636764160,"available":95887880192,"internally_reserved":0,"allocated":309825536,"data_stored":309645567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,6,0,2,2,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726364,"kb_used_data":302272,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93645476,"statfs":{"total":96636764160,"available":95892967424,"internally_reserved":0,"allocated":309526528,"data_stored":309346559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,1,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557408,"kb_used_data":259216,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93814432,"statfs":{"total":96636764160,"available":96065978368,"internally_reserved":0,"allocated":265437184,"data_stored":265257215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":305327826},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728620,"kb_used_data":299924,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93643220,"statfs":{"total":96636764160,"available":95890657280,"internally_reserved":0,"allocated":307122176,"data_stored":306942207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,2,0,4,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306475008,"data_stored":306475008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265138176,"data_stored":265138176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308854784,"data_stored":308854784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":309518336,"data_stored":309518336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:35.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:35.904 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-11T05:01:35.749411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":610625160,"num_objects":74545,"num_object_clones":0,"num_object_copies":298180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120648,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126864,"ondisk_log_size":126864,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747144,"kb_used_data":1154940,"kb_used_omap":14,"kb_used_meta":1592113,"kb_avail":374740216,"statfs":{"total":386547056640,"available":383733981184,"internally_reserved":0,"allocated":1182658560,"data_stored":1181938684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1630324278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,10,6,13,10,1,0,3,0,0,0,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24559616,"num_objects":-2998,"num_object_clones":0,"num_object_copies":-11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2998,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004542"},"pg_stats":[{"pgid":"1.f","version":"33'7906","reported_seq":7981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.486477+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:34.486477+0000","last_peered":"2024-09-11T05:01:34.486477+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:34.486477+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7906,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8000","reported_seq":8095,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.989868+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:32.989868+0000","last_peered":"2024-09-11T05:01:32.989868+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:32.989868+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810784,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2473}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7685","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.418930+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:35.418930+0000","last_peered":"2024-09-11T05:01:35.418930+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:35.418930+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7685,"log_dups_size":0,"ondisk_log_size":7685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36657768,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7684,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7788","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.934108+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:32.934108+0000","last_peered":"2024-09-11T05:01:32.934108+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:32.934108+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37909664,"num_objects":4628,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7786,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7824","reported_seq":7914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.665920+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:34.665920+0000","last_peered":"2024-09-11T05:01:34.665920+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:34.665920+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38556832,"num_objects":4707,"num_object_clones":0,"num_object_copies":18828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7822,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7994","reported_seq":8082,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.544749+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:34.544749+0000","last_peered":"2024-09-11T05:01:34.544749+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:34.544749+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7994,"log_dups_size":0,"ondisk_log_size":7994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38060032,"num_objects":4646,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7994,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7956","reported_seq":8034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.985948+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:32.985948+0000","last_peered":"2024-09-11T05:01:32.985948+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:32.985948+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7956,"log_dups_size":0,"ondisk_log_size":7956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39105696,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2462}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8092","reported_seq":10698,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:33.067033+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:33.067033+0000","last_peered":"2024-09-11T05:01:33.067033+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:33.067033+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38769824,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1271,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8136","reported_seq":8237,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.599856+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:34.599856+0000","last_peered":"2024-09-11T05:01:34.599856+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:34.599856+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38103360,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7937","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.359486+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:35.359486+0000","last_peered":"2024-09-11T05:01:35.359486+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:35.359486+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37606560,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2285}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7985","reported_seq":8078,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.438484+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:34.438484+0000","last_peered":"2024-09-11T05:01:34.438484+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:34.438484+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38235888,"num_objects":4668,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7982,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7714","reported_seq":12309,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:33.903345+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:33.903345+0000","last_peered":"2024-09-11T05:01:33.903345+0000","last_clean":"2024-09-11T05:01:33.903345+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:33.903345+0000","last_undegraded":"2024-09-11T05:01:33.903345+0000","last_fullsized":"2024-09-11T05:01:33.903345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7714,"log_dups_size":0,"ondisk_log_size":7714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38256640,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8033","reported_seq":8124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.515778+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:35.515778+0000","last_peered":"2024-09-11T05:01:35.515778+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:35.515778+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38419024,"num_objects":4690,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7949","reported_seq":8034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.366015+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:34.366015+0000","last_peered":"2024-09-11T05:01:34.366015+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:34.366015+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7949,"log_dups_size":0,"ondisk_log_size":7949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38032544,"num_objects":4643,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7947,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2371}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7852","reported_seq":7961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.360515+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:35.360515+0000","last_peered":"2024-09-11T05:01:35.360515+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:35.360515+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328032,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8013","reported_seq":8103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:33.012161+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:33.012161+0000","last_peered":"2024-09-11T05:01:33.012161+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:33.012161+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":610625160,"num_objects":74545,"num_object_clones":0,"num_object_copies":298180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120648,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1180573696,"data_stored":1180573696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126864,"ondisk_log_size":126864,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731080,"kb_used_data":298164,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93640760,"statfs":{"total":96636764160,"available":95888138240,"internally_reserved":0,"allocated":305319936,"data_stored":305139967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5,1,1,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726100,"kb_used_data":297860,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93645740,"statfs":{"total":96636764160,"available":95893237760,"internally_reserved":0,"allocated":305008640,"data_stored":304828671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,0,4,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561344,"kb_used_data":258992,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93810496,"statfs":{"total":96636764160,"available":96061947904,"internally_reserved":0,"allocated":265207808,"data_stored":265027839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728620,"kb_used_data":299924,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93643220,"statfs":{"total":96636764160,"available":95890657280,"internally_reserved":0,"allocated":307122176,"data_stored":306942207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,2,0,4,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306475008,"data_stored":306475008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264912896,"data_stored":264912896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304226304,"data_stored":304226304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304959488,"data_stored":304959488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:35.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:35.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7906", 'reported_seq': 7981, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:34.486477+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:34.486477+0000', 'last_peered': '2024-09-11T05:01:34.486477+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:34.486477+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7906, 'log_dups_size': 0, 'ondisk_log_size': 7906, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37748736, 'num_objects': 4608, 'num_object_clones': 0, 'num_object_copies': 18432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7906, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2310}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:35.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:35.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8000", 'reported_seq': 8095, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:32.989868+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:32.989868+0000', 'last_peered': '2024-09-11T05:01:32.989868+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:32.989868+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8000, 'log_dups_size': 0, 'ondisk_log_size': 8000, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38810784, 'num_objects': 4738, 'num_object_clones': 0, 'num_object_copies': 18952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2267, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7998, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2473}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:35.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:35.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7685", 'reported_seq': 7780, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:35.418930+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:35.418930+0000', 'last_peered': '2024-09-11T05:01:35.418930+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:35.418930+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7685, 'log_dups_size': 0, 'ondisk_log_size': 7685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36657768, 'num_objects': 4476, 'num_object_clones': 0, 'num_object_copies': 17904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4476, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7684, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2263}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7788", 'reported_seq': 7871, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:32.934108+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:32.934108+0000', 'last_peered': '2024-09-11T05:01:32.934108+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:32.934108+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7788, 'log_dups_size': 0, 'ondisk_log_size': 7788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37909664, 'num_objects': 4628, 'num_object_clones': 0, 'num_object_copies': 18512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7786, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2357}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7824", 'reported_seq': 7914, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:34.665920+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:34.665920+0000', 'last_peered': '2024-09-11T05:01:34.665920+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:34.665920+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7824, 'log_dups_size': 0, 'ondisk_log_size': 7824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38556832, 'num_objects': 4707, 'num_object_clones': 0, 'num_object_copies': 18828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7822, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2472}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7994", 'reported_seq': 8082, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:34.544749+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:34.544749+0000', 'last_peered': '2024-09-11T05:01:34.544749+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:34.544749+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7994, 'log_dups_size': 0, 'ondisk_log_size': 7994, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38060032, 'num_objects': 4646, 'num_object_clones': 0, 'num_object_copies': 18584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2300, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7994, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2346}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7956", 'reported_seq': 8034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:32.985948+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:32.985948+0000', 'last_peered': '2024-09-11T05:01:32.985948+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:32.985948+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7956, 'log_dups_size': 0, 'ondisk_log_size': 7956, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39105696, 'num_objects': 4774, 'num_object_clones': 0, 'num_object_copies': 19096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7954, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2462}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:35.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8092", 'reported_seq': 10698, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:33.067033+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:33.067033+0000', 'last_peered': '2024-09-11T05:01:33.067033+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:33.067033+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8092, 'log_dups_size': 0, 'ondisk_log_size': 8092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38769824, 'num_objects': 4733, 'num_object_clones': 0, 'num_object_copies': 18932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1094, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8090, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1271, 'num_bytes_recovered': 10338304, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8136", 'reported_seq': 8237, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:34.599856+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:34.599856+0000', 'last_peered': '2024-09-11T05:01:34.599856+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:34.599856+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8136, 'log_dups_size': 0, 'ondisk_log_size': 8136, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38103360, 'num_objects': 4652, 'num_object_clones': 0, 'num_object_copies': 18608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2377, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8133, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7937", 'reported_seq': 8044, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:35.359486+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:35.359486+0000', 'last_peered': '2024-09-11T05:01:35.359486+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:35.359486+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7937, 'log_dups_size': 0, 'ondisk_log_size': 7937, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37606560, 'num_objects': 4591, 'num_object_clones': 0, 'num_object_copies': 18364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7935, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2285}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7985", 'reported_seq': 8078, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:34.438484+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:34.438484+0000', 'last_peered': '2024-09-11T05:01:34.438484+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:34.438484+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7985, 'log_dups_size': 0, 'ondisk_log_size': 7985, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38235888, 'num_objects': 4668, 'num_object_clones': 0, 'num_object_copies': 18672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7982, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2361}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8033", 'reported_seq': 8124, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:35.515778+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:35.515778+0000', 'last_peered': '2024-09-11T05:01:35.515778+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:35.515778+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8033, 'log_dups_size': 0, 'ondisk_log_size': 8033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38419024, 'num_objects': 4690, 'num_object_clones': 0, 'num_object_copies': 18760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8032, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7949", 'reported_seq': 8034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:34.366015+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:34.366015+0000', 'last_peered': '2024-09-11T05:01:34.366015+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:34.366015+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7949, 'log_dups_size': 0, 'ondisk_log_size': 7949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38032544, 'num_objects': 4643, 'num_object_clones': 0, 'num_object_copies': 18572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7947, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2371}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:35.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7852", 'reported_seq': 7961, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:35.360515+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:35.360515+0000', 'last_peered': '2024-09-11T05:01:35.360515+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:35.360515+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7852, 'log_dups_size': 0, 'ondisk_log_size': 7852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37328032, 'num_objects': 4557, 'num_object_clones': 0, 'num_object_copies': 18228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7850, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:35.908 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8013", 'reported_seq': 8103, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:33.012161+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:33.012161+0000', 'last_peered': '2024-09-11T05:01:33.012161+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:33.012161+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8013, 'log_dups_size': 0, 'ondisk_log_size': 8013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39023776, 'num_objects': 4764, 'num_object_clones': 0, 'num_object_copies': 19056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2302, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8011, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:35.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:36.652 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:36.653 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33202/298180 objects degraded (11.135%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":423,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74545,"data_bytes":610625160,"bytes_used":2813075456,"bytes_avail":383733981184,"bytes_total":386547056640,"degraded_objects":33202,"degraded_total":298180,"degraded_ratio":0.11134884968810783,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":82571,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":249},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:36.691 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:36.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:37.287 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:37.288 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:37.326 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-11T05:01:35.749411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":610625160,"num_objects":74545,"num_object_clones":0,"num_object_copies":298180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120648,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126864,"ondisk_log_size":126864,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747144,"kb_used_data":1154940,"kb_used_omap":14,"kb_used_meta":1592113,"kb_avail":374740216,"statfs":{"total":386547056640,"available":383733981184,"internally_reserved":0,"allocated":1182658560,"data_stored":1181938684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1630324278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,10,6,13,10,1,0,3,0,0,0,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24559616,"num_objects":-2998,"num_object_clones":0,"num_object_copies":-11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2998,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004542"},"pg_stats":[{"pgid":"1.f","version":"33'7906","reported_seq":7981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.486477+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:34.486477+0000","last_peered":"2024-09-11T05:01:34.486477+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:34.486477+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7906,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8000","reported_seq":8095,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.989868+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:32.989868+0000","last_peered":"2024-09-11T05:01:32.989868+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:32.989868+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810784,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2473}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7685","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.418930+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:35.418930+0000","last_peered":"2024-09-11T05:01:35.418930+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:35.418930+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7685,"log_dups_size":0,"ondisk_log_size":7685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36657768,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7684,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7788","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.934108+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:32.934108+0000","last_peered":"2024-09-11T05:01:32.934108+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:32.934108+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37909664,"num_objects":4628,"num_object_clones":0,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7786,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7824","reported_seq":7914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.665920+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:34.665920+0000","last_peered":"2024-09-11T05:01:34.665920+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:34.665920+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38556832,"num_objects":4707,"num_object_clones":0,"num_object_copies":18828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7822,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7994","reported_seq":8082,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.544749+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:34.544749+0000","last_peered":"2024-09-11T05:01:34.544749+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:34.544749+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7994,"log_dups_size":0,"ondisk_log_size":7994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38060032,"num_objects":4646,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7994,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7956","reported_seq":8034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:32.985948+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:32.985948+0000","last_peered":"2024-09-11T05:01:32.985948+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:32.985948+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7956,"log_dups_size":0,"ondisk_log_size":7956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39105696,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2462}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8092","reported_seq":10698,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:33.067033+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:33.067033+0000","last_peered":"2024-09-11T05:01:33.067033+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:33.067033+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38769824,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1271,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8136","reported_seq":8237,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.599856+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:34.599856+0000","last_peered":"2024-09-11T05:01:34.599856+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:34.599856+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38103360,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7937","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.359486+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:35.359486+0000","last_peered":"2024-09-11T05:01:35.359486+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:35.359486+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37606560,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2285}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7985","reported_seq":8078,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.438484+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:34.438484+0000","last_peered":"2024-09-11T05:01:34.438484+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:34.438484+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38235888,"num_objects":4668,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7982,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7714","reported_seq":12309,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:33.903345+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:33.903345+0000","last_peered":"2024-09-11T05:01:33.903345+0000","last_clean":"2024-09-11T05:01:33.903345+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:33.903345+0000","last_undegraded":"2024-09-11T05:01:33.903345+0000","last_fullsized":"2024-09-11T05:01:33.903345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7714,"log_dups_size":0,"ondisk_log_size":7714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38256640,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8033","reported_seq":8124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.515778+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:35.515778+0000","last_peered":"2024-09-11T05:01:35.515778+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:35.515778+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38419024,"num_objects":4690,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7949","reported_seq":8034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:34.366015+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:34.366015+0000","last_peered":"2024-09-11T05:01:34.366015+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:34.366015+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7949,"log_dups_size":0,"ondisk_log_size":7949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38032544,"num_objects":4643,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7947,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2371}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7852","reported_seq":7961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.360515+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:35.360515+0000","last_peered":"2024-09-11T05:01:35.360515+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:35.360515+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328032,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8013","reported_seq":8103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:33.012161+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:33.012161+0000","last_peered":"2024-09-11T05:01:33.012161+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:33.012161+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39023776,"num_objects":4764,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":610625160,"num_objects":74545,"num_object_clones":0,"num_object_copies":298180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120648,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":10338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1180573696,"data_stored":1180573696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126864,"ondisk_log_size":126864,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731080,"kb_used_data":298164,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93640760,"statfs":{"total":96636764160,"available":95888138240,"internally_reserved":0,"allocated":305319936,"data_stored":305139967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5,1,1,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726100,"kb_used_data":297860,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93645740,"statfs":{"total":96636764160,"available":95893237760,"internally_reserved":0,"allocated":305008640,"data_stored":304828671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,0,4,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561344,"kb_used_data":258992,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93810496,"statfs":{"total":96636764160,"available":96061947904,"internally_reserved":0,"allocated":265207808,"data_stored":265027839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728620,"kb_used_data":299924,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93643220,"statfs":{"total":96636764160,"available":95890657280,"internally_reserved":0,"allocated":307122176,"data_stored":306942207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,2,0,4,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306475008,"data_stored":306475008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264912896,"data_stored":264912896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304226304,"data_stored":304226304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304959488,"data_stored":304959488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:40.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:40.992 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:41.015 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-11T05:01:39.751263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603719304,"num_objects":73702,"num_object_clones":0,"num_object_copies":294808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121491,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1548,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127707,"ondisk_log_size":127707,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2750300,"kb_used_data":1145588,"kb_used_omap":14,"kb_used_meta":1604593,"kb_avail":374737060,"statfs":{"total":386547056640,"available":383730749440,"internally_reserved":0,"allocated":1173082112,"data_stored":1172362236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1643103798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,7,19,5,19,1,0,0,5,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25223168,"num_objects":-3079,"num_object_clones":0,"num_object_copies":-12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004794"},"pg_stats":[{"pgid":"1.f","version":"33'7973","reported_seq":8054,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.481020+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:39.481020+0000","last_peered":"2024-09-11T05:01:39.481020+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:39.481020+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7973,"log_dups_size":0,"ondisk_log_size":7973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37199872,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8072","reported_seq":8172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.166701+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:38.166701+0000","last_peered":"2024-09-11T05:01:38.166701+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:38.166701+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38220960,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7685","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.418930+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:35.418930+0000","last_peered":"2024-09-11T05:01:35.418930+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:35.418930+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7685,"log_dups_size":0,"ondisk_log_size":7685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36657768,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7684,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7866","reported_seq":7959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:37.926804+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:37.926804+0000","last_peered":"2024-09-11T05:01:37.926804+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:37.926804+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270688,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7864,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7890","reported_seq":7990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.677776+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:39.677776+0000","last_peered":"2024-09-11T05:01:39.677776+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:39.677776+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7890,"log_dups_size":0,"ondisk_log_size":7890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38016160,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8057","reported_seq":8147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.630734+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:39.630734+0000","last_peered":"2024-09-11T05:01:39.630734+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:39.630734+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37543936,"num_objects":4583,"num_object_clones":0,"num_object_copies":18332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8057,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8030","reported_seq":8110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.104192+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:38.104192+0000","last_peered":"2024-09-11T05:01:38.104192+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:38.104192+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38499488,"num_objects":4700,"num_object_clones":0,"num_object_copies":18800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8170","reported_seq":10874,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.125590+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:38.125590+0000","last_peered":"2024-09-11T05:01:38.125590+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:38.125590+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38130848,"num_objects":4655,"num_object_clones":0,"num_object_copies":18620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8209","reported_seq":8314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.569650+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:39.569650+0000","last_peered":"2024-09-11T05:01:39.569650+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:39.569650+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37505344,"num_objects":4579,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8206,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7937","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.359486+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:35.359486+0000","last_peered":"2024-09-11T05:01:35.359486+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:35.359486+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37606560,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2285}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8054","reported_seq":8150,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.615413+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:39.615413+0000","last_peered":"2024-09-11T05:01:39.615413+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:39.615413+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8054,"log_dups_size":0,"ondisk_log_size":8054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37670640,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7784","reported_seq":12379,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:39.069015+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:39.069015+0000","last_peered":"2024-09-11T05:01:39.069015+0000","last_clean":"2024-09-11T05:01:39.069015+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:39.069015+0000","last_undegraded":"2024-09-11T05:01:39.069015+0000","last_fullsized":"2024-09-11T05:01:39.069015+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37683200,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1594,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8033","reported_seq":8124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.515778+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:35.515778+0000","last_peered":"2024-09-11T05:01:35.515778+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:35.515778+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38419024,"num_objects":4690,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8005","reported_seq":8098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.421077+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:39.421077+0000","last_peered":"2024-09-11T05:01:39.421077+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:39.421077+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2319}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7852","reported_seq":7961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.360515+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:35.360515+0000","last_peered":"2024-09-11T05:01:35.360515+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:35.360515+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328032,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8090","reported_seq":8184,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.115693+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:38.115693+0000","last_peered":"2024-09-11T05:01:38.115693+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:38.115693+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8090,"log_dups_size":0,"ondisk_log_size":8090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38392992,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8088,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603719304,"num_objects":73702,"num_object_clones":0,"num_object_copies":294808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121491,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1548,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1170993152,"data_stored":1170993152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127707,"ondisk_log_size":127707,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730700,"kb_used_data":293600,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93641140,"statfs":{"total":96636764160,"available":95888527360,"internally_reserved":0,"allocated":300646400,"data_stored":300466431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,1,5,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726100,"kb_used_data":297860,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93645740,"statfs":{"total":96636764160,"available":95893237760,"internally_reserved":0,"allocated":305008640,"data_stored":304828671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,0,4,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565248,"kb_used_data":258736,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93806592,"statfs":{"total":96636764160,"available":96057950208,"internally_reserved":0,"allocated":264945664,"data_stored":264765695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313847506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728252,"kb_used_data":295392,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93643588,"statfs":{"total":96636764160,"available":95891034112,"internally_reserved":0,"allocated":302481408,"data_stored":302301439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,9,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301821952,"data_stored":301821952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264650752,"data_stored":264650752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304226304,"data_stored":304226304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300294144,"data_stored":300294144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:41.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:41.662 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-11T05:01:39.751263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603719304,"num_objects":73702,"num_object_clones":0,"num_object_copies":294808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121491,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1548,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127707,"ondisk_log_size":127707,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2750300,"kb_used_data":1145588,"kb_used_omap":14,"kb_used_meta":1604593,"kb_avail":374737060,"statfs":{"total":386547056640,"available":383730749440,"internally_reserved":0,"allocated":1173082112,"data_stored":1172362236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1643103798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,7,19,5,19,1,0,0,5,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25223168,"num_objects":-3079,"num_object_clones":0,"num_object_copies":-12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004794"},"pg_stats":[{"pgid":"1.f","version":"33'7973","reported_seq":8054,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.481020+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:39.481020+0000","last_peered":"2024-09-11T05:01:39.481020+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:39.481020+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7973,"log_dups_size":0,"ondisk_log_size":7973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37199872,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8072","reported_seq":8172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.166701+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:38.166701+0000","last_peered":"2024-09-11T05:01:38.166701+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:38.166701+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38220960,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7685","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.418930+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:35.418930+0000","last_peered":"2024-09-11T05:01:35.418930+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:35.418930+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7685,"log_dups_size":0,"ondisk_log_size":7685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36657768,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7684,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7866","reported_seq":7959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:37.926804+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:37.926804+0000","last_peered":"2024-09-11T05:01:37.926804+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:37.926804+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270688,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7864,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7890","reported_seq":7990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.677776+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:39.677776+0000","last_peered":"2024-09-11T05:01:39.677776+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:39.677776+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7890,"log_dups_size":0,"ondisk_log_size":7890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38016160,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8057","reported_seq":8147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.630734+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:39.630734+0000","last_peered":"2024-09-11T05:01:39.630734+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:39.630734+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37543936,"num_objects":4583,"num_object_clones":0,"num_object_copies":18332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8057,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8030","reported_seq":8110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.104192+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:38.104192+0000","last_peered":"2024-09-11T05:01:38.104192+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:38.104192+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38499488,"num_objects":4700,"num_object_clones":0,"num_object_copies":18800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8170","reported_seq":10874,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.125590+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:38.125590+0000","last_peered":"2024-09-11T05:01:38.125590+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:38.125590+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38130848,"num_objects":4655,"num_object_clones":0,"num_object_copies":18620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8209","reported_seq":8314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.569650+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:39.569650+0000","last_peered":"2024-09-11T05:01:39.569650+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:39.569650+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37505344,"num_objects":4579,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8206,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7937","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.359486+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:35.359486+0000","last_peered":"2024-09-11T05:01:35.359486+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:35.359486+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37606560,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2285}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8054","reported_seq":8150,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.615413+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:39.615413+0000","last_peered":"2024-09-11T05:01:39.615413+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:39.615413+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8054,"log_dups_size":0,"ondisk_log_size":8054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37670640,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7784","reported_seq":12379,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:39.069015+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:39.069015+0000","last_peered":"2024-09-11T05:01:39.069015+0000","last_clean":"2024-09-11T05:01:39.069015+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:39.069015+0000","last_undegraded":"2024-09-11T05:01:39.069015+0000","last_fullsized":"2024-09-11T05:01:39.069015+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37683200,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1594,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8033","reported_seq":8124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.515778+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:35.515778+0000","last_peered":"2024-09-11T05:01:35.515778+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:35.515778+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38419024,"num_objects":4690,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8005","reported_seq":8098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.421077+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:39.421077+0000","last_peered":"2024-09-11T05:01:39.421077+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:39.421077+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2319}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7852","reported_seq":7961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:35.360515+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:35.360515+0000","last_peered":"2024-09-11T05:01:35.360515+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:35.360515+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328032,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8090","reported_seq":8184,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.115693+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:38.115693+0000","last_peered":"2024-09-11T05:01:38.115693+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:38.115693+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8090,"log_dups_size":0,"ondisk_log_size":8090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38392992,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8088,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603719304,"num_objects":73702,"num_object_clones":0,"num_object_copies":294808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121491,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1548,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1170993152,"data_stored":1170993152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127707,"ondisk_log_size":127707,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730700,"kb_used_data":293600,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93641140,"statfs":{"total":96636764160,"available":95888527360,"internally_reserved":0,"allocated":300646400,"data_stored":300466431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,1,5,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726100,"kb_used_data":297860,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93645740,"statfs":{"total":96636764160,"available":95893237760,"internally_reserved":0,"allocated":305008640,"data_stored":304828671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,0,4,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565248,"kb_used_data":258736,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93806592,"statfs":{"total":96636764160,"available":96057950208,"internally_reserved":0,"allocated":264945664,"data_stored":264765695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313847506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728252,"kb_used_data":295392,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93643588,"statfs":{"total":96636764160,"available":95891034112,"internally_reserved":0,"allocated":302481408,"data_stored":302301439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,9,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301821952,"data_stored":301821952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264650752,"data_stored":264650752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304226304,"data_stored":304226304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300294144,"data_stored":300294144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:41.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:41.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7973", 'reported_seq': 8054, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:39.481020+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:39.481020+0000', 'last_peered': '2024-09-11T05:01:39.481020+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:39.481020+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7973, 'log_dups_size': 0, 'ondisk_log_size': 7973, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37199872, 'num_objects': 4541, 'num_object_clones': 0, 'num_object_copies': 18164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7973, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:41.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:41.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8072", 'reported_seq': 8172, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:38.166701+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:38.166701+0000', 'last_peered': '2024-09-11T05:01:38.166701+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:38.166701+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8072, 'log_dups_size': 0, 'ondisk_log_size': 8072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38220960, 'num_objects': 4666, 'num_object_clones': 0, 'num_object_copies': 18664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8070, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7685", 'reported_seq': 7780, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:35.418930+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:35.418930+0000', 'last_peered': '2024-09-11T05:01:35.418930+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:35.418930+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7685, 'log_dups_size': 0, 'ondisk_log_size': 7685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36657768, 'num_objects': 4476, 'num_object_clones': 0, 'num_object_copies': 17904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4476, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7684, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2263}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7866", 'reported_seq': 7959, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:37.926804+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:37.926804+0000', 'last_peered': '2024-09-11T05:01:37.926804+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:37.926804+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7866, 'log_dups_size': 0, 'ondisk_log_size': 7866, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37270688, 'num_objects': 4550, 'num_object_clones': 0, 'num_object_copies': 18200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7864, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2284}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7890", 'reported_seq': 7990, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:39.677776+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:39.677776+0000', 'last_peered': '2024-09-11T05:01:39.677776+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:39.677776+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7890, 'log_dups_size': 0, 'ondisk_log_size': 7890, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38016160, 'num_objects': 4641, 'num_object_clones': 0, 'num_object_copies': 18564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7888, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2411}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8057", 'reported_seq': 8147, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:39.630734+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:39.630734+0000', 'last_peered': '2024-09-11T05:01:39.630734+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:39.630734+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8057, 'log_dups_size': 0, 'ondisk_log_size': 8057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37543936, 'num_objects': 4583, 'num_object_clones': 0, 'num_object_copies': 18332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2299, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8057, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2284}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:41.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8030", 'reported_seq': 8110, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:38.104192+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:38.104192+0000', 'last_peered': '2024-09-11T05:01:38.104192+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:38.104192+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8030, 'log_dups_size': 0, 'ondisk_log_size': 8030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38499488, 'num_objects': 4700, 'num_object_clones': 0, 'num_object_copies': 18800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8028, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2389}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8170", 'reported_seq': 10874, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:38.125590+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:38.125590+0000', 'last_peered': '2024-09-11T05:01:38.125590+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:38.125590+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8170, 'log_dups_size': 0, 'ondisk_log_size': 8170, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38130848, 'num_objects': 4655, 'num_object_clones': 0, 'num_object_copies': 18620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8168, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1320, 'num_bytes_recovered': 10739712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3610}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8209", 'reported_seq': 8314, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:39.569650+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:39.569650+0000', 'last_peered': '2024-09-11T05:01:39.569650+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:39.569650+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8209, 'log_dups_size': 0, 'ondisk_log_size': 8209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37505344, 'num_objects': 4579, 'num_object_clones': 0, 'num_object_copies': 18316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2375, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8206, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2207}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7937", 'reported_seq': 8044, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:35.359486+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:35.359486+0000', 'last_peered': '2024-09-11T05:01:35.359486+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:35.359486+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7937, 'log_dups_size': 0, 'ondisk_log_size': 7937, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37606560, 'num_objects': 4591, 'num_object_clones': 0, 'num_object_copies': 18364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7935, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2285}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8054", 'reported_seq': 8150, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:39.615413+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:39.615413+0000', 'last_peered': '2024-09-11T05:01:39.615413+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:39.615413+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8054, 'log_dups_size': 0, 'ondisk_log_size': 8054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37670640, 'num_objects': 4599, 'num_object_clones': 0, 'num_object_copies': 18396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8051, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:41.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:41.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8033", 'reported_seq': 8124, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:35.515778+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:35.515778+0000', 'last_peered': '2024-09-11T05:01:35.515778+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:35.515778+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8033, 'log_dups_size': 0, 'ondisk_log_size': 8033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38419024, 'num_objects': 4690, 'num_object_clones': 0, 'num_object_copies': 18760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8032, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:41.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:41.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8005", 'reported_seq': 8098, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:39.421077+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:39.421077+0000', 'last_peered': '2024-09-11T05:01:39.421077+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:39.421077+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8005, 'log_dups_size': 0, 'ondisk_log_size': 8005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37573792, 'num_objects': 4587, 'num_object_clones': 0, 'num_object_copies': 18348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8003, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2319}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:41.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:41.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7852", 'reported_seq': 7961, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:35.360515+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:35.360515+0000', 'last_peered': '2024-09-11T05:01:35.360515+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:35.360515+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7852, 'log_dups_size': 0, 'ondisk_log_size': 7852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37328032, 'num_objects': 4557, 'num_object_clones': 0, 'num_object_copies': 18228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7850, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:41.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:41.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8090", 'reported_seq': 8184, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:38.115693+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:38.115693+0000', 'last_peered': '2024-09-11T05:01:38.115693+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:38.115693+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8090, 'log_dups_size': 0, 'ondisk_log_size': 8090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38392992, 'num_objects': 4687, 'num_object_clones': 0, 'num_object_copies': 18748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2300, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8088, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:41.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:42.415 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:42.415 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33124/293592 objects degraded (11.282%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":429,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73398,"data_bytes":601228936,"bytes_used":2815909888,"bytes_avail":383731146752,"bytes_total":386547056640,"degraded_objects":33124,"degraded_total":293592,"degraded_ratio":0.11282323769040029,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":76428,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":917504,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:42.442 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:42.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:43.057 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:43.081 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-11T05:01:41.752195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601228936,"num_objects":73398,"num_object_clones":0,"num_object_copies":293592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121795,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128011,"ondisk_log_size":128011,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749912,"kb_used_data":1141036,"kb_used_omap":14,"kb_used_meta":1608753,"kb_avail":374737448,"statfs":{"total":386547056640,"available":383731146752,"internally_reserved":0,"allocated":1168420864,"data_stored":1167700988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1647363638},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,2,6,19,6,15,1,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20865024,"num_objects":-2547,"num_object_clones":0,"num_object_copies":-10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004807"},"pg_stats":[{"pgid":"1.f","version":"33'7973","reported_seq":8054,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.481020+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:39.481020+0000","last_peered":"2024-09-11T05:01:39.481020+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:39.481020+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7973,"log_dups_size":0,"ondisk_log_size":7973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37199872,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8072","reported_seq":8172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.166701+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:38.166701+0000","last_peered":"2024-09-11T05:01:38.166701+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:38.166701+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38220960,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7754","reported_seq":7849,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:40.347922+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:40.347922+0000","last_peered":"2024-09-11T05:01:40.347922+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:40.347922+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36092520,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7753,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7866","reported_seq":7959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:37.926804+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:37.926804+0000","last_peered":"2024-09-11T05:01:37.926804+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:37.926804+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270688,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7864,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7890","reported_seq":7990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.677776+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:39.677776+0000","last_peered":"2024-09-11T05:01:39.677776+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:39.677776+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7890,"log_dups_size":0,"ondisk_log_size":7890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38016160,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8057","reported_seq":8147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.630734+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:39.630734+0000","last_peered":"2024-09-11T05:01:39.630734+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:39.630734+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37543936,"num_objects":4583,"num_object_clones":0,"num_object_copies":18332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8057,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8030","reported_seq":8110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.104192+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:38.104192+0000","last_peered":"2024-09-11T05:01:38.104192+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:38.104192+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38499488,"num_objects":4700,"num_object_clones":0,"num_object_copies":18800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8170","reported_seq":10874,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.125590+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:38.125590+0000","last_peered":"2024-09-11T05:01:38.125590+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:38.125590+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38130848,"num_objects":4655,"num_object_clones":0,"num_object_copies":18620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8209","reported_seq":8314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.569650+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:39.569650+0000","last_peered":"2024-09-11T05:01:39.569650+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:39.569650+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37505344,"num_objects":4579,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8206,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8014","reported_seq":8121,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:40.280374+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:40.280374+0000","last_peered":"2024-09-11T05:01:40.280374+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:40.280374+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8014,"log_dups_size":0,"ondisk_log_size":8014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36975776,"num_objects":4514,"num_object_clones":0,"num_object_copies":18056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8012,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2208}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8054","reported_seq":8150,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.615413+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:39.615413+0000","last_peered":"2024-09-11T05:01:39.615413+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:39.615413+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8054,"log_dups_size":0,"ondisk_log_size":8054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37670640,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7784","reported_seq":12379,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:39.069015+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:39.069015+0000","last_peered":"2024-09-11T05:01:39.069015+0000","last_clean":"2024-09-11T05:01:39.069015+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:39.069015+0000","last_undegraded":"2024-09-11T05:01:39.069015+0000","last_fullsized":"2024-09-11T05:01:39.069015+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37683200,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1594,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8112","reported_seq":8203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:40.473739+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:40.473739+0000","last_peered":"2024-09-11T05:01:40.473739+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:40.473739+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8112,"log_dups_size":0,"ondisk_log_size":8112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37771856,"num_objects":4611,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2285}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8005","reported_seq":8098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:39.421077+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:39.421077+0000","last_peered":"2024-09-11T05:01:39.421077+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:39.421077+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573792,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2319}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7931","reported_seq":8042,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:40.445342+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:40.445342+0000","last_peered":"2024-09-11T05:01:40.445342+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:40.445342+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36680864,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7929,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8090","reported_seq":8184,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:38.115693+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:38.115693+0000","last_peered":"2024-09-11T05:01:38.115693+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:38.115693+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8090,"log_dups_size":0,"ondisk_log_size":8090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38392992,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8088,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":601228936,"num_objects":73398,"num_object_clones":0,"num_object_copies":293592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121795,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":10739712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1166286848,"data_stored":1166286848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128011,"ondisk_log_size":128011,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730700,"kb_used_data":293600,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93641140,"statfs":{"total":96636764160,"available":95888527360,"internally_reserved":0,"allocated":300646400,"data_stored":300466431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,1,5,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725712,"kb_used_data":293308,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93646128,"statfs":{"total":96636764160,"available":95893635072,"internally_reserved":0,"allocated":300347392,"data_stored":300167423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565248,"kb_used_data":258736,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93806592,"statfs":{"total":96636764160,"available":96057950208,"internally_reserved":0,"allocated":264945664,"data_stored":264765695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313847506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728252,"kb_used_data":295392,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93643588,"statfs":{"total":96636764160,"available":95891034112,"internally_reserved":0,"allocated":302481408,"data_stored":302301439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,9,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301821952,"data_stored":301821952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264650752,"data_stored":264650752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299520000,"data_stored":299520000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300294144,"data_stored":300294144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:46.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:46.719 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:46.720 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:46.744 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-11T05:01:45.753954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592160392,"num_objects":72291,"num_object_clones":0,"num_object_copies":289164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122902,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1621,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129118,"ondisk_log_size":129118,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790708,"kb_used_data":1127696,"kb_used_omap":14,"kb_used_meta":1662897,"kb_avail":374696652,"statfs":{"total":386547056640,"available":383689371648,"internally_reserved":0,"allocated":1154760704,"data_stored":1154040828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1702807094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,3,3,16,13,1,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25010176,"num_objects":-3053,"num_object_clones":0,"num_object_copies":-12212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005242"},"pg_stats":[{"pgid":"1.f","version":"33'8046","reported_seq":8133,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.655121+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:44.655121+0000","last_peered":"2024-09-11T05:01:44.655121+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:44.655121+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36601856,"num_objects":4468,"num_object_clones":0,"num_object_copies":17872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8130","reported_seq":8233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:42.944249+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:42.944249+0000","last_peered":"2024-09-11T05:01:42.944249+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:42.944249+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37745824,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8128,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7811","reported_seq":7909,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.482614+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:45.482614+0000","last_peered":"2024-09-11T05:01:45.482614+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:45.482614+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35625576,"num_objects":4350,"num_object_clones":0,"num_object_copies":17400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7810,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2138}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7932","reported_seq":8029,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.020791+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:43.020791+0000","last_peered":"2024-09-11T05:01:43.020791+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:43.020791+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36730016,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7966","reported_seq":8076,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.622836+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:44.622836+0000","last_peered":"2024-09-11T05:01:44.622836+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:44.622836+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7966,"log_dups_size":0,"ondisk_log_size":7966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393568,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7964,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8141","reported_seq":8233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.573148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:44.573148+0000","last_peered":"2024-09-11T05:01:44.573148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:44.573148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36855808,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8096","reported_seq":8177,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.034645+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:43.034645+0000","last_peered":"2024-09-11T05:01:43.034645+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:43.034645+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37958816,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8236","reported_seq":11052,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.150500+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:43.150500+0000","last_peered":"2024-09-11T05:01:43.150500+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:43.150500+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590176,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3600}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8271","reported_seq":8378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.678394+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:44.678394+0000","last_peered":"2024-09-11T05:01:44.678394+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:44.678394+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8271,"log_dups_size":0,"ondisk_log_size":8271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36997440,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8268,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8095","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.485185+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:45.485185+0000","last_peered":"2024-09-11T05:01:45.485185+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:45.485185+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8095,"log_dups_size":0,"ondisk_log_size":8095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36312224,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8093,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2127}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8106","reported_seq":8205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.494403+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:44.494403+0000","last_peered":"2024-09-11T05:01:44.494403+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:44.494403+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8106,"log_dups_size":0,"ondisk_log_size":8106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37244656,"num_objects":4547,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7856","reported_seq":12451,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:43.909384+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:43.909384+0000","last_peered":"2024-09-11T05:01:43.909384+0000","last_clean":"2024-09-11T05:01:43.909384+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:43.909384+0000","last_undegraded":"2024-09-11T05:01:43.909384+0000","last_fullsized":"2024-09-11T05:01:43.909384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37093376,"num_objects":4528,"num_object_clones":0,"num_object_copies":18112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1666,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8181","reported_seq":8272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.479373+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:45.479373+0000","last_peered":"2024-09-11T05:01:45.479373+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:45.479373+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37206608,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8180,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8082","reported_seq":8175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.412616+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:44.412616+0000","last_peered":"2024-09-11T05:01:44.412616+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:44.412616+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36943008,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8005","reported_seq":8116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.555534+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:45.555534+0000","last_peered":"2024-09-11T05:01:45.555534+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:45.555534+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074656,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8164","reported_seq":8260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:42.944059+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:42.944059+0000","last_peered":"2024-09-11T05:01:42.944059+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:42.944059+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37786784,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8162,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":592160392,"num_objects":72291,"num_object_clones":0,"num_object_copies":289164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122902,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1621,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1152393216,"data_stored":1152393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129118,"ondisk_log_size":129118,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732016,"kb_used_data":289368,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93639824,"statfs":{"total":96636764160,"available":95887179776,"internally_reserved":0,"allocated":296312832,"data_stored":296132863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":453242576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,0,3,3,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743464,"kb_used_data":288844,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93628376,"statfs":{"total":96636764160,"available":95875457024,"internally_reserved":0,"allocated":295776256,"data_stored":295596287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569164,"kb_used_data":258492,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93802676,"statfs":{"total":96636764160,"available":96053940224,"internally_reserved":0,"allocated":264695808,"data_stored":264515839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318107346},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746064,"kb_used_data":290992,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93625776,"statfs":{"total":96636764160,"available":95872794624,"internally_reserved":0,"allocated":297975808,"data_stored":297795839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":465956559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297267200,"data_stored":297267200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264396800,"data_stored":264396800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294936576,"data_stored":294936576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":295792640,"data_stored":295792640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:46.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:47.330 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:47.331 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:47.360 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-11T05:01:45.753954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592160392,"num_objects":72291,"num_object_clones":0,"num_object_copies":289164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122902,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1621,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129118,"ondisk_log_size":129118,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790708,"kb_used_data":1127696,"kb_used_omap":14,"kb_used_meta":1662897,"kb_avail":374696652,"statfs":{"total":386547056640,"available":383689371648,"internally_reserved":0,"allocated":1154760704,"data_stored":1154040828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1702807094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,3,3,16,13,1,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25010176,"num_objects":-3053,"num_object_clones":0,"num_object_copies":-12212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005242"},"pg_stats":[{"pgid":"1.f","version":"33'8046","reported_seq":8133,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.655121+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:44.655121+0000","last_peered":"2024-09-11T05:01:44.655121+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:44.655121+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36601856,"num_objects":4468,"num_object_clones":0,"num_object_copies":17872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8130","reported_seq":8233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:42.944249+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:42.944249+0000","last_peered":"2024-09-11T05:01:42.944249+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:42.944249+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37745824,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8128,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7811","reported_seq":7909,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.482614+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:45.482614+0000","last_peered":"2024-09-11T05:01:45.482614+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:45.482614+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35625576,"num_objects":4350,"num_object_clones":0,"num_object_copies":17400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7810,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2138}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7932","reported_seq":8029,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.020791+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:43.020791+0000","last_peered":"2024-09-11T05:01:43.020791+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:43.020791+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36730016,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7966","reported_seq":8076,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.622836+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:44.622836+0000","last_peered":"2024-09-11T05:01:44.622836+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:44.622836+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7966,"log_dups_size":0,"ondisk_log_size":7966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393568,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7964,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8141","reported_seq":8233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.573148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:44.573148+0000","last_peered":"2024-09-11T05:01:44.573148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:44.573148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36855808,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8096","reported_seq":8177,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.034645+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:43.034645+0000","last_peered":"2024-09-11T05:01:43.034645+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:43.034645+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37958816,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8236","reported_seq":11052,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.150500+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:43.150500+0000","last_peered":"2024-09-11T05:01:43.150500+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:43.150500+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590176,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3600}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8271","reported_seq":8378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.678394+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:44.678394+0000","last_peered":"2024-09-11T05:01:44.678394+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:44.678394+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8271,"log_dups_size":0,"ondisk_log_size":8271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36997440,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8268,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8095","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.485185+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:45.485185+0000","last_peered":"2024-09-11T05:01:45.485185+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:45.485185+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8095,"log_dups_size":0,"ondisk_log_size":8095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36312224,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8093,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2127}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8106","reported_seq":8205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.494403+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:44.494403+0000","last_peered":"2024-09-11T05:01:44.494403+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:44.494403+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8106,"log_dups_size":0,"ondisk_log_size":8106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37244656,"num_objects":4547,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7856","reported_seq":12451,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:43.909384+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:43.909384+0000","last_peered":"2024-09-11T05:01:43.909384+0000","last_clean":"2024-09-11T05:01:43.909384+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:43.909384+0000","last_undegraded":"2024-09-11T05:01:43.909384+0000","last_fullsized":"2024-09-11T05:01:43.909384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37093376,"num_objects":4528,"num_object_clones":0,"num_object_copies":18112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1666,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8181","reported_seq":8272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.479373+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:45.479373+0000","last_peered":"2024-09-11T05:01:45.479373+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:45.479373+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37206608,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8180,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8082","reported_seq":8175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.412616+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:44.412616+0000","last_peered":"2024-09-11T05:01:44.412616+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:44.412616+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36943008,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8005","reported_seq":8116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.555534+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:45.555534+0000","last_peered":"2024-09-11T05:01:45.555534+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:45.555534+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074656,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8164","reported_seq":8260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:42.944059+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:42.944059+0000","last_peered":"2024-09-11T05:01:42.944059+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:42.944059+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37786784,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8162,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":592160392,"num_objects":72291,"num_object_clones":0,"num_object_copies":289164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122902,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1621,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1152393216,"data_stored":1152393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129118,"ondisk_log_size":129118,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732016,"kb_used_data":289368,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93639824,"statfs":{"total":96636764160,"available":95887179776,"internally_reserved":0,"allocated":296312832,"data_stored":296132863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":453242576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,0,3,3,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743464,"kb_used_data":288844,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93628376,"statfs":{"total":96636764160,"available":95875457024,"internally_reserved":0,"allocated":295776256,"data_stored":295596287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569164,"kb_used_data":258492,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93802676,"statfs":{"total":96636764160,"available":96053940224,"internally_reserved":0,"allocated":264695808,"data_stored":264515839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318107346},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746064,"kb_used_data":290992,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93625776,"statfs":{"total":96636764160,"available":95872794624,"internally_reserved":0,"allocated":297975808,"data_stored":297795839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":465956559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297267200,"data_stored":297267200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264396800,"data_stored":264396800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294936576,"data_stored":294936576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":295792640,"data_stored":295792640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:47.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8046", 'reported_seq': 8133, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:44.655121+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:44.655121+0000', 'last_peered': '2024-09-11T05:01:44.655121+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:44.655121+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8046, 'log_dups_size': 0, 'ondisk_log_size': 8046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36601856, 'num_objects': 4468, 'num_object_clones': 0, 'num_object_copies': 17872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8046, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2176}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8130", 'reported_seq': 8233, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:42.944249+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:42.944249+0000', 'last_peered': '2024-09-11T05:01:42.944249+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:42.944249+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8130, 'log_dups_size': 0, 'ondisk_log_size': 8130, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37745824, 'num_objects': 4608, 'num_object_clones': 0, 'num_object_copies': 18432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8128, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2347}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7811", 'reported_seq': 7909, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:45.482614+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:45.482614+0000', 'last_peered': '2024-09-11T05:01:45.482614+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:45.482614+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7811, 'log_dups_size': 0, 'ondisk_log_size': 7811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35625576, 'num_objects': 4350, 'num_object_clones': 0, 'num_object_copies': 17400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4350, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7810, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2138}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7932", 'reported_seq': 8029, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:43.020791+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:43.020791+0000', 'last_peered': '2024-09-11T05:01:43.020791+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:43.020791+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7932, 'log_dups_size': 0, 'ondisk_log_size': 7932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36730016, 'num_objects': 4484, 'num_object_clones': 0, 'num_object_copies': 17936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7930, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:47.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7966", 'reported_seq': 8076, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:44.622836+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:44.622836+0000', 'last_peered': '2024-09-11T05:01:44.622836+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:44.622836+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7966, 'log_dups_size': 0, 'ondisk_log_size': 7966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37393568, 'num_objects': 4565, 'num_object_clones': 0, 'num_object_copies': 18260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7964, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8141", 'reported_seq': 8233, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:44.573148+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:44.573148+0000', 'last_peered': '2024-09-11T05:01:44.573148+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:44.573148+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8141, 'log_dups_size': 0, 'ondisk_log_size': 8141, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36855808, 'num_objects': 4499, 'num_object_clones': 0, 'num_object_copies': 17996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8141, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8096", 'reported_seq': 8177, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:43.034645+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:43.034645+0000', 'last_peered': '2024-09-11T05:01:43.034645+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:43.034645+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8096, 'log_dups_size': 0, 'ondisk_log_size': 8096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37958816, 'num_objects': 4634, 'num_object_clones': 0, 'num_object_copies': 18536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8094, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2323}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8236", 'reported_seq': 11052, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:43.150500+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:43.150500+0000', 'last_peered': '2024-09-11T05:01:43.150500+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:43.150500+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8236, 'log_dups_size': 0, 'ondisk_log_size': 8236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37590176, 'num_objects': 4589, 'num_object_clones': 0, 'num_object_copies': 18356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8234, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1376, 'num_bytes_recovered': 11198464, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3600}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8271", 'reported_seq': 8378, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:44.678394+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:44.678394+0000', 'last_peered': '2024-09-11T05:01:44.678394+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:44.678394+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8271, 'log_dups_size': 0, 'ondisk_log_size': 8271, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36997440, 'num_objects': 4517, 'num_object_clones': 0, 'num_object_copies': 18068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2374, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8268, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2146}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8095", 'reported_seq': 8202, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:45.485185+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:45.485185+0000', 'last_peered': '2024-09-11T05:01:45.485185+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:45.485185+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8095, 'log_dups_size': 0, 'ondisk_log_size': 8095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36312224, 'num_objects': 4433, 'num_object_clones': 0, 'num_object_copies': 17732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8093, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2127}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:47.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8106", 'reported_seq': 8205, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:44.494403+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:44.494403+0000', 'last_peered': '2024-09-11T05:01:44.494403+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:44.494403+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8106, 'log_dups_size': 0, 'ondisk_log_size': 8106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37244656, 'num_objects': 4547, 'num_object_clones': 0, 'num_object_copies': 18188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8103, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2243}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8181", 'reported_seq': 8272, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:45.479373+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:45.479373+0000', 'last_peered': '2024-09-11T05:01:45.479373+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:45.479373+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8181, 'log_dups_size': 0, 'ondisk_log_size': 8181, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37206608, 'num_objects': 4542, 'num_object_clones': 0, 'num_object_copies': 18168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8180, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8082", 'reported_seq': 8175, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:44.412616+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:44.412616+0000', 'last_peered': '2024-09-11T05:01:44.412616+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:44.412616+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8082, 'log_dups_size': 0, 'ondisk_log_size': 8082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36943008, 'num_objects': 4510, 'num_object_clones': 0, 'num_object_copies': 18040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8080, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8005", 'reported_seq': 8116, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:45.555534+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:45.555534+0000', 'last_peered': '2024-09-11T05:01:45.555534+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:45.555534+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8005, 'log_dups_size': 0, 'ondisk_log_size': 8005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36074656, 'num_objects': 4404, 'num_object_clones': 0, 'num_object_copies': 17616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8003, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:47.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8164", 'reported_seq': 8260, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:42.944059+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:42.944059+0000', 'last_peered': '2024-09-11T05:01:42.944059+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:42.944059+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8164, 'log_dups_size': 0, 'ondisk_log_size': 8164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37786784, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 18452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2299, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8162, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2316}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:47.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:48.181 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:48.182 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33052/289164 objects degraded (11.430%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":435,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72291,"data_bytes":592160392,"bytes_used":2857684992,"bytes_avail":383689371648,"bytes_total":386547056640,"degraded_objects":33052,"degraded_total":289164,"degraded_ratio":0.11430191863440814,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":71647,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":860160,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":187},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:48.214 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:48.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:48.845 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:48.875 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-11T05:01:47.754919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592160392,"num_objects":72291,"num_object_clones":0,"num_object_copies":289164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122902,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1621,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129118,"ondisk_log_size":129118,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790708,"kb_used_data":1127696,"kb_used_omap":14,"kb_used_meta":1662897,"kb_avail":374696652,"statfs":{"total":386547056640,"available":383689371648,"internally_reserved":0,"allocated":1154760704,"data_stored":1154040828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1702807094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,3,3,16,13,1,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18464768,"num_objects":-2254,"num_object_clones":0,"num_object_copies":-9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2254,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005507"},"pg_stats":[{"pgid":"1.f","version":"33'8046","reported_seq":8133,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.655121+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:44.655121+0000","last_peered":"2024-09-11T05:01:44.655121+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:44.655121+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36601856,"num_objects":4468,"num_object_clones":0,"num_object_copies":17872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8130","reported_seq":8233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:42.944249+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:42.944249+0000","last_peered":"2024-09-11T05:01:42.944249+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:42.944249+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37745824,"num_objects":4608,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8128,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7811","reported_seq":7909,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.482614+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:45.482614+0000","last_peered":"2024-09-11T05:01:45.482614+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:45.482614+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35625576,"num_objects":4350,"num_object_clones":0,"num_object_copies":17400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7810,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2138}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7932","reported_seq":8029,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.020791+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:43.020791+0000","last_peered":"2024-09-11T05:01:43.020791+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:43.020791+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36730016,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7966","reported_seq":8076,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.622836+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:44.622836+0000","last_peered":"2024-09-11T05:01:44.622836+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:44.622836+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7966,"log_dups_size":0,"ondisk_log_size":7966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393568,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7964,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8141","reported_seq":8233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.573148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:44.573148+0000","last_peered":"2024-09-11T05:01:44.573148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:44.573148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36855808,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8096","reported_seq":8177,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.034645+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:43.034645+0000","last_peered":"2024-09-11T05:01:43.034645+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:43.034645+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37958816,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8236","reported_seq":11052,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:43.150500+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:43.150500+0000","last_peered":"2024-09-11T05:01:43.150500+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:43.150500+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590176,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3600}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8271","reported_seq":8378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.678394+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:44.678394+0000","last_peered":"2024-09-11T05:01:44.678394+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:44.678394+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8271,"log_dups_size":0,"ondisk_log_size":8271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36997440,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8268,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8095","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.485185+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:45.485185+0000","last_peered":"2024-09-11T05:01:45.485185+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:45.485185+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8095,"log_dups_size":0,"ondisk_log_size":8095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36312224,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8093,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2127}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8106","reported_seq":8205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.494403+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:44.494403+0000","last_peered":"2024-09-11T05:01:44.494403+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:44.494403+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8106,"log_dups_size":0,"ondisk_log_size":8106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37244656,"num_objects":4547,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7856","reported_seq":12451,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:43.909384+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:43.909384+0000","last_peered":"2024-09-11T05:01:43.909384+0000","last_clean":"2024-09-11T05:01:43.909384+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:43.909384+0000","last_undegraded":"2024-09-11T05:01:43.909384+0000","last_fullsized":"2024-09-11T05:01:43.909384+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37093376,"num_objects":4528,"num_object_clones":0,"num_object_copies":18112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1666,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8181","reported_seq":8272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.479373+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:45.479373+0000","last_peered":"2024-09-11T05:01:45.479373+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:45.479373+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37206608,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8180,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8082","reported_seq":8175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:44.412616+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:44.412616+0000","last_peered":"2024-09-11T05:01:44.412616+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:44.412616+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36943008,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8005","reported_seq":8116,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:45.555534+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:45.555534+0000","last_peered":"2024-09-11T05:01:45.555534+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:45.555534+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074656,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8164","reported_seq":8260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:42.944059+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:42.944059+0000","last_peered":"2024-09-11T05:01:42.944059+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:42.944059+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37786784,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8162,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":592160392,"num_objects":72291,"num_object_clones":0,"num_object_copies":289164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122902,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1621,"num_bytes_recovered":11198464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1152393216,"data_stored":1152393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129118,"ondisk_log_size":129118,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732016,"kb_used_data":289368,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93639824,"statfs":{"total":96636764160,"available":95887179776,"internally_reserved":0,"allocated":296312832,"data_stored":296132863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":453242576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,0,3,3,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743464,"kb_used_data":288844,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93628376,"statfs":{"total":96636764160,"available":95875457024,"internally_reserved":0,"allocated":295776256,"data_stored":295596287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569164,"kb_used_data":258492,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93802676,"statfs":{"total":96636764160,"available":96053940224,"internally_reserved":0,"allocated":264695808,"data_stored":264515839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":318107346},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746064,"kb_used_data":290992,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93625776,"statfs":{"total":96636764160,"available":95872794624,"internally_reserved":0,"allocated":297975808,"data_stored":297795839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":465956559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297267200,"data_stored":297267200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264396800,"data_stored":264396800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294936576,"data_stored":294936576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":295792640,"data_stored":295792640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:51.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:52.493 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:52.513 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-11T05:01:51.756785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583050888,"num_objects":71179,"num_object_clones":0,"num_object_copies":284716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124014,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":11722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130230,"ondisk_log_size":130230,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2764816,"kb_used_data":1114304,"kb_used_omap":14,"kb_used_meta":1650417,"kb_avail":374722544,"statfs":{"total":386547056640,"available":383715885056,"internally_reserved":0,"allocated":1141047296,"data_stored":1140327420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1690027574},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,2,9,3,3,2,0,0,4,0,1,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20668416,"num_objects":-2523,"num_object_clones":0,"num_object_copies":-10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2523,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005521"},"pg_stats":[{"pgid":"1.f","version":"33'8112","reported_seq":8198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.753178+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:49.753178+0000","last_peered":"2024-09-11T05:01:49.753178+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:49.753178+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8112,"log_dups_size":0,"ondisk_log_size":8112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36061184,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2110}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8196","reported_seq":8299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:48.157975+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:48.157975+0000","last_peered":"2024-09-11T05:01:48.157975+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:48.157975+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8196,"log_dups_size":0,"ondisk_log_size":8196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37205152,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7880","reported_seq":7978,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.614184+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:50.614184+0000","last_peered":"2024-09-11T05:01:50.614184+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:50.614184+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7880,"log_dups_size":0,"ondisk_log_size":7880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35060328,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7879,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2069}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7995","reported_seq":8092,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:48.074777+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:48.074777+0000","last_peered":"2024-09-11T05:01:48.074777+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:48.074777+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213920,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8039","reported_seq":8148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.769291+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:49.769291+0000","last_peered":"2024-09-11T05:01:49.769291+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:49.769291+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8039,"log_dups_size":0,"ondisk_log_size":8039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36795552,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8037,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8203","reported_seq":8296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.659672+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:49.659672+0000","last_peered":"2024-09-11T05:01:49.659672+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:49.659672+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8203,"log_dups_size":0,"ondisk_log_size":8203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36347904,"num_objects":4437,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8203,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8167","reported_seq":8248,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:48.013437+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:48.013437+0000","last_peered":"2024-09-11T05:01:48.013437+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:48.013437+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8167,"log_dups_size":0,"ondisk_log_size":8167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37377184,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2252}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8305","reported_seq":11249,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:48.154838+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:48.154838+0000","last_peered":"2024-09-11T05:01:48.154838+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:48.154838+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8305,"log_dups_size":0,"ondisk_log_size":8305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37024928,"num_objects":4520,"num_object_clones":0,"num_object_copies":18080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":11722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8347","reported_seq":8464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.805740+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:49.805740+0000","last_peered":"2024-09-11T05:01:49.805740+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:49.805740+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36374848,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8157","reported_seq":8264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.544745+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:50.544745+0000","last_peered":"2024-09-11T05:01:50.544745+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:50.544745+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8157,"log_dups_size":0,"ondisk_log_size":8157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804320,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8155,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8176","reported_seq":8275,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.751508+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:49.751508+0000","last_peered":"2024-09-11T05:01:49.751508+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:49.751508+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8176,"log_dups_size":0,"ondisk_log_size":8176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36671216,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8173,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7939","reported_seq":12533,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:49.077442+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:49.077442+0000","last_peered":"2024-09-11T05:01:49.077442+0000","last_clean":"2024-09-11T05:01:49.077442+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:49.077442+0000","last_undegraded":"2024-09-11T05:01:49.077442+0000","last_fullsized":"2024-09-11T05:01:49.077442+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36413440,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1749,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8258","reported_seq":8348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.634904+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:50.634904+0000","last_peered":"2024-09-11T05:01:50.634904+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:50.634904+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36575824,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8257,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2139}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8154","reported_seq":8247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.664954+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:49.664954+0000","last_peered":"2024-09-11T05:01:49.664954+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:49.664954+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8154,"log_dups_size":0,"ondisk_log_size":8154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36353184,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8068","reported_seq":8178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.489883+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:50.489883+0000","last_peered":"2024-09-11T05:01:50.489883+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:50.489883+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8068,"log_dups_size":0,"ondisk_log_size":8068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35558560,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8234","reported_seq":8330,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:47.883042+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:47.883042+0000","last_peered":"2024-09-11T05:01:47.883042+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:47.883042+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8234,"log_dups_size":0,"ondisk_log_size":8234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37213344,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8232,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583050888,"num_objects":71179,"num_object_clones":0,"num_object_copies":284716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124014,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":11722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138520064,"data_stored":1138520064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130230,"ondisk_log_size":130230,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731820,"kb_used_data":285004,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93640020,"statfs":{"total":96636764160,"available":95887380480,"internally_reserved":0,"allocated":291844096,"data_stored":291664127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":457502416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,3,2,2,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726644,"kb_used_data":284516,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93645196,"statfs":{"total":96636764160,"available":95892680704,"internally_reserved":0,"allocated":291344384,"data_stored":291164415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560524,"kb_used_data":258172,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93811316,"statfs":{"total":96636764160,"available":96062787584,"internally_reserved":0,"allocated":264368128,"data_stored":264188159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745828,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":459195,"kb_avail":93626012,"statfs":{"total":96636764160,"available":95873036288,"internally_reserved":0,"allocated":293490688,"data_stored":293310719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":470216399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292720640,"data_stored":292720640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264093696,"data_stored":264093696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290443264,"data_stored":290443264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291262464,"data_stored":291262464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:52.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:52.983 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:52.984 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:53.014 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-11T05:01:51.756785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583050888,"num_objects":71179,"num_object_clones":0,"num_object_copies":284716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124014,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":11722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130230,"ondisk_log_size":130230,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2764816,"kb_used_data":1114304,"kb_used_omap":14,"kb_used_meta":1650417,"kb_avail":374722544,"statfs":{"total":386547056640,"available":383715885056,"internally_reserved":0,"allocated":1141047296,"data_stored":1140327420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1690027574},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,2,9,3,3,2,0,0,4,0,1,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20668416,"num_objects":-2523,"num_object_clones":0,"num_object_copies":-10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2523,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005521"},"pg_stats":[{"pgid":"1.f","version":"33'8112","reported_seq":8198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.753178+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:49.753178+0000","last_peered":"2024-09-11T05:01:49.753178+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:49.753178+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8112,"log_dups_size":0,"ondisk_log_size":8112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36061184,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2110}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8196","reported_seq":8299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:48.157975+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:48.157975+0000","last_peered":"2024-09-11T05:01:48.157975+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:48.157975+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8196,"log_dups_size":0,"ondisk_log_size":8196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37205152,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7880","reported_seq":7978,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.614184+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:50.614184+0000","last_peered":"2024-09-11T05:01:50.614184+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:50.614184+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7880,"log_dups_size":0,"ondisk_log_size":7880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35060328,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7879,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2069}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7995","reported_seq":8092,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:48.074777+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:48.074777+0000","last_peered":"2024-09-11T05:01:48.074777+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:48.074777+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213920,"num_objects":4421,"num_object_clones":0,"num_object_copies":17684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8039","reported_seq":8148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.769291+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:49.769291+0000","last_peered":"2024-09-11T05:01:49.769291+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:49.769291+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8039,"log_dups_size":0,"ondisk_log_size":8039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36795552,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8037,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8203","reported_seq":8296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.659672+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:49.659672+0000","last_peered":"2024-09-11T05:01:49.659672+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:49.659672+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8203,"log_dups_size":0,"ondisk_log_size":8203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36347904,"num_objects":4437,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8203,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8167","reported_seq":8248,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:48.013437+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:48.013437+0000","last_peered":"2024-09-11T05:01:48.013437+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:48.013437+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8167,"log_dups_size":0,"ondisk_log_size":8167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37377184,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2252}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8305","reported_seq":11249,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:48.154838+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:48.154838+0000","last_peered":"2024-09-11T05:01:48.154838+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:48.154838+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8305,"log_dups_size":0,"ondisk_log_size":8305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37024928,"num_objects":4520,"num_object_clones":0,"num_object_copies":18080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":11722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8347","reported_seq":8464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.805740+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:49.805740+0000","last_peered":"2024-09-11T05:01:49.805740+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:49.805740+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36374848,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8157","reported_seq":8264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.544745+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:50.544745+0000","last_peered":"2024-09-11T05:01:50.544745+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:50.544745+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8157,"log_dups_size":0,"ondisk_log_size":8157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804320,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8155,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8176","reported_seq":8275,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.751508+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:49.751508+0000","last_peered":"2024-09-11T05:01:49.751508+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:49.751508+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8176,"log_dups_size":0,"ondisk_log_size":8176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36671216,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8173,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7939","reported_seq":12533,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:49.077442+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:49.077442+0000","last_peered":"2024-09-11T05:01:49.077442+0000","last_clean":"2024-09-11T05:01:49.077442+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:49.077442+0000","last_undegraded":"2024-09-11T05:01:49.077442+0000","last_fullsized":"2024-09-11T05:01:49.077442+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36413440,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1749,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8258","reported_seq":8348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.634904+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:50.634904+0000","last_peered":"2024-09-11T05:01:50.634904+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:50.634904+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36575824,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8257,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2139}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8154","reported_seq":8247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.664954+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:49.664954+0000","last_peered":"2024-09-11T05:01:49.664954+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:49.664954+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8154,"log_dups_size":0,"ondisk_log_size":8154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36353184,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8068","reported_seq":8178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.489883+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:50.489883+0000","last_peered":"2024-09-11T05:01:50.489883+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:50.489883+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8068,"log_dups_size":0,"ondisk_log_size":8068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35558560,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8234","reported_seq":8330,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:47.883042+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:47.883042+0000","last_peered":"2024-09-11T05:01:47.883042+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:47.883042+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8234,"log_dups_size":0,"ondisk_log_size":8234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37213344,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8232,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583050888,"num_objects":71179,"num_object_clones":0,"num_object_copies":284716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124014,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":11722752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138520064,"data_stored":1138520064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130230,"ondisk_log_size":130230,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731820,"kb_used_data":285004,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93640020,"statfs":{"total":96636764160,"available":95887380480,"internally_reserved":0,"allocated":291844096,"data_stored":291664127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":457502416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,3,2,2,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726644,"kb_used_data":284516,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93645196,"statfs":{"total":96636764160,"available":95892680704,"internally_reserved":0,"allocated":291344384,"data_stored":291164415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560524,"kb_used_data":258172,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93811316,"statfs":{"total":96636764160,"available":96062787584,"internally_reserved":0,"allocated":264368128,"data_stored":264188159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745828,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":459195,"kb_avail":93626012,"statfs":{"total":96636764160,"available":95873036288,"internally_reserved":0,"allocated":293490688,"data_stored":293310719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":470216399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292720640,"data_stored":292720640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264093696,"data_stored":264093696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290443264,"data_stored":290443264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291262464,"data_stored":291262464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:53.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:53.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8112", 'reported_seq': 8198, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:49.753178+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:49.753178+0000', 'last_peered': '2024-09-11T05:01:49.753178+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:49.753178+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8112, 'log_dups_size': 0, 'ondisk_log_size': 8112, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36061184, 'num_objects': 4402, 'num_object_clones': 0, 'num_object_copies': 17608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8112, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2110}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:53.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:53.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8196", 'reported_seq': 8299, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:48.157975+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:48.157975+0000', 'last_peered': '2024-09-11T05:01:48.157975+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:48.157975+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8196, 'log_dups_size': 0, 'ondisk_log_size': 8196, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37205152, 'num_objects': 4542, 'num_object_clones': 0, 'num_object_copies': 18168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8194, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:53.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:53.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7880", 'reported_seq': 7978, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:50.614184+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:50.614184+0000', 'last_peered': '2024-09-11T05:01:50.614184+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:50.614184+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7880, 'log_dups_size': 0, 'ondisk_log_size': 7880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35060328, 'num_objects': 4281, 'num_object_clones': 0, 'num_object_copies': 17124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4281, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7879, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2069}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:53.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:53.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7995", 'reported_seq': 8092, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:48.074777+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:48.074777+0000', 'last_peered': '2024-09-11T05:01:48.074777+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:48.074777+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7995, 'log_dups_size': 0, 'ondisk_log_size': 7995, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36213920, 'num_objects': 4421, 'num_object_clones': 0, 'num_object_copies': 17684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7993, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:53.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:53.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8039", 'reported_seq': 8148, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:49.769291+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:49.769291+0000', 'last_peered': '2024-09-11T05:01:49.769291+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:49.769291+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8039, 'log_dups_size': 0, 'ondisk_log_size': 8039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36795552, 'num_objects': 4492, 'num_object_clones': 0, 'num_object_copies': 17968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8037, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2267}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:53.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:53.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8203", 'reported_seq': 8296, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:49.659672+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:49.659672+0000', 'last_peered': '2024-09-11T05:01:49.659672+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:49.659672+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8203, 'log_dups_size': 0, 'ondisk_log_size': 8203, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36347904, 'num_objects': 4437, 'num_object_clones': 0, 'num_object_copies': 17748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8203, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2140}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:53.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:53.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8167", 'reported_seq': 8248, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:48.013437+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:48.013437+0000', 'last_peered': '2024-09-11T05:01:48.013437+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:48.013437+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8167, 'log_dups_size': 0, 'ondisk_log_size': 8167, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37377184, 'num_objects': 4563, 'num_object_clones': 0, 'num_object_copies': 18252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8165, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2252}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:53.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:53.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8305", 'reported_seq': 11249, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:48.154838+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:48.154838+0000', 'last_peered': '2024-09-11T05:01:48.154838+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:48.154838+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8305, 'log_dups_size': 0, 'ondisk_log_size': 8305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37024928, 'num_objects': 4520, 'num_object_clones': 0, 'num_object_copies': 18080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 925, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8303, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1440, 'num_bytes_recovered': 11722752, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3595}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:53.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:53.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8347", 'reported_seq': 8464, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:49.805740+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:49.805740+0000', 'last_peered': '2024-09-11T05:01:49.805740+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:49.805740+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8347, 'log_dups_size': 0, 'ondisk_log_size': 8347, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36374848, 'num_objects': 4441, 'num_object_clones': 0, 'num_object_copies': 17764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8344, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2075}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:53.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:53.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8157", 'reported_seq': 8264, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:50.544745+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:50.544745+0000', 'last_peered': '2024-09-11T05:01:50.544745+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:50.544745+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8157, 'log_dups_size': 0, 'ondisk_log_size': 8157, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35804320, 'num_objects': 4371, 'num_object_clones': 0, 'num_object_copies': 17484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8155, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2065}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:53.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:53.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8176", 'reported_seq': 8275, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:49.751508+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:49.751508+0000', 'last_peered': '2024-09-11T05:01:49.751508+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:49.751508+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8176, 'log_dups_size': 0, 'ondisk_log_size': 8176, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36671216, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 17908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2307, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8173, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2173}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:53.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:53.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8258", 'reported_seq': 8348, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:50.634904+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:50.634904+0000', 'last_peered': '2024-09-11T05:01:50.634904+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:50.634904+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8258, 'log_dups_size': 0, 'ondisk_log_size': 8258, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36575824, 'num_objects': 4465, 'num_object_clones': 0, 'num_object_copies': 17860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8257, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2139}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:53.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:53.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8154", 'reported_seq': 8247, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:49.664954+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:49.664954+0000', 'last_peered': '2024-09-11T05:01:49.664954+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:49.664954+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8154, 'log_dups_size': 0, 'ondisk_log_size': 8154, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36353184, 'num_objects': 4438, 'num_object_clones': 0, 'num_object_copies': 17752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8152, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2170}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:53.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:53.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8068", 'reported_seq': 8178, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:50.489883+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:50.489883+0000', 'last_peered': '2024-09-11T05:01:50.489883+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:50.489883+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8068, 'log_dups_size': 0, 'ondisk_log_size': 8068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35558560, 'num_objects': 4341, 'num_object_clones': 0, 'num_object_copies': 17364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8066, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:53.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:53.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8234", 'reported_seq': 8330, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:47.883042+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:47.883042+0000', 'last_peered': '2024-09-11T05:01:47.883042+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:47.883042+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8234, 'log_dups_size': 0, 'ondisk_log_size': 8234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37213344, 'num_objects': 4543, 'num_object_clones': 0, 'num_object_copies': 18172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2299, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8232, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:53.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:53.810 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:53.811 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32982/284716 objects degraded (11.584%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":440,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71179,"data_bytes":583050888,"bytes_used":2831171584,"bytes_avail":383715885056,"bytes_total":386547056640,"degraded_objects":32982,"degraded_total":284716,"degraded_ratio":0.11584175107826747,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":81882,"recovering_keys_per_sec":0,"num_objects_recovered":143,"num_bytes_recovered":983040,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:53.834 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:53.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:54.486 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:54.508 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-11T05:01:53.757770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":580519560,"num_objects":70870,"num_object_clones":0,"num_object_copies":283480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124323,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1775,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130539,"ondisk_log_size":130539,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765988,"kb_used_data":1109908,"kb_used_omap":14,"kb_used_meta":1655985,"kb_avail":374721372,"statfs":{"total":386547056640,"available":383714684928,"internally_reserved":0,"allocated":1136545792,"data_stored":1135825916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1695729206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,3,10,6,4,1,0,0,4,0,1,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20709376,"num_objects":-2528,"num_object_clones":0,"num_object_copies":-10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005575"},"pg_stats":[{"pgid":"1.f","version":"33'8112","reported_seq":8198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.753178+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:49.753178+0000","last_peered":"2024-09-11T05:01:49.753178+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:49.753178+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8112,"log_dups_size":0,"ondisk_log_size":8112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36061184,"num_objects":4402,"num_object_clones":0,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2110}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8261","reported_seq":8364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.011061+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:53.011061+0000","last_peered":"2024-09-11T05:01:53.011061+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:53.011061+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672672,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7880","reported_seq":7978,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.614184+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:50.614184+0000","last_peered":"2024-09-11T05:01:50.614184+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:50.614184+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7880,"log_dups_size":0,"ondisk_log_size":7880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35060328,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7879,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2069}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8057","reported_seq":8154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.129679+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:53.129679+0000","last_peered":"2024-09-11T05:01:53.129679+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:53.129679+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35706016,"num_objects":4359,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8055,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2095}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8039","reported_seq":8148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.769291+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:49.769291+0000","last_peered":"2024-09-11T05:01:49.769291+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:49.769291+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8039,"log_dups_size":0,"ondisk_log_size":8039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36795552,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8037,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8203","reported_seq":8296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.659672+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:49.659672+0000","last_peered":"2024-09-11T05:01:49.659672+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:49.659672+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8203,"log_dups_size":0,"ondisk_log_size":8203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36347904,"num_objects":4437,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8203,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8232","reported_seq":8313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.134392+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:53.134392+0000","last_peered":"2024-09-11T05:01:53.134392+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:53.134392+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36844704,"num_objects":4498,"num_object_clones":0,"num_object_copies":17992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8361","reported_seq":11472,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.149575+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:53.149575+0000","last_peered":"2024-09-11T05:01:53.149575+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:53.149575+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36566176,"num_objects":4464,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8347","reported_seq":8464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.805740+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:49.805740+0000","last_peered":"2024-09-11T05:01:49.805740+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:49.805740+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36374848,"num_objects":4441,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8157","reported_seq":8264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.544745+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:50.544745+0000","last_peered":"2024-09-11T05:01:50.544745+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:50.544745+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8157,"log_dups_size":0,"ondisk_log_size":8157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804320,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8155,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8176","reported_seq":8275,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.751508+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:49.751508+0000","last_peered":"2024-09-11T05:01:49.751508+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:49.751508+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8176,"log_dups_size":0,"ondisk_log_size":8176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36671216,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8173,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7939","reported_seq":12533,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:49.077442+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:49.077442+0000","last_peered":"2024-09-11T05:01:49.077442+0000","last_clean":"2024-09-11T05:01:49.077442+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:49.077442+0000","last_undegraded":"2024-09-11T05:01:49.077442+0000","last_fullsized":"2024-09-11T05:01:49.077442+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36413440,"num_objects":4445,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1749,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8258","reported_seq":8348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.634904+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:50.634904+0000","last_peered":"2024-09-11T05:01:50.634904+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:50.634904+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36575824,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8257,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2139}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8154","reported_seq":8247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:49.664954+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:49.664954+0000","last_peered":"2024-09-11T05:01:49.664954+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:49.664954+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8154,"log_dups_size":0,"ondisk_log_size":8154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36353184,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8068","reported_seq":8178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:50.489883+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:50.489883+0000","last_peered":"2024-09-11T05:01:50.489883+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:50.489883+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8068,"log_dups_size":0,"ondisk_log_size":8068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35558560,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8295","reported_seq":8392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.219287+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:53.219287+0000","last_peered":"2024-09-11T05:01:53.219287+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:53.219287+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8295,"log_dups_size":0,"ondisk_log_size":8295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36713632,"num_objects":4482,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8293,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":580519560,"num_objects":70870,"num_object_clones":0,"num_object_copies":283480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124323,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1775,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1134059520,"data_stored":1134059520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130539,"ondisk_log_size":130539,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731820,"kb_used_data":285004,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93640020,"statfs":{"total":96636764160,"available":95887380480,"internally_reserved":0,"allocated":291844096,"data_stored":291664127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":457502416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,3,2,2,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726644,"kb_used_data":284516,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93645196,"statfs":{"total":96636764160,"available":95892680704,"internally_reserved":0,"allocated":291344384,"data_stored":291164415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560524,"kb_used_data":258172,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93811316,"statfs":{"total":96636764160,"available":96062787584,"internally_reserved":0,"allocated":264368128,"data_stored":264188159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":309587666},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747000,"kb_used_data":282216,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93624840,"statfs":{"total":96636764160,"available":95871836160,"internally_reserved":0,"allocated":288989184,"data_stored":288809215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,1,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288260096,"data_stored":288260096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264093696,"data_stored":264093696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290443264,"data_stored":290443264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291262464,"data_stored":291262464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:57.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:58.107 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:58.107 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:58.132 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-11T05:01:57.759674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":574367368,"num_objects":70119,"num_object_clones":0,"num_object_copies":280476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125074,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1782,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131290,"ondisk_log_size":131290,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805576,"kb_used_data":1100928,"kb_used_omap":14,"kb_used_meta":1704561,"kb_avail":374681784,"statfs":{"total":386547056640,"available":383674146816,"internally_reserved":0,"allocated":1127350272,"data_stored":1126630396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1745471030},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,7,7,17,4,0,0,0,0,1,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17793024,"num_objects":-2172,"num_object_clones":0,"num_object_copies":-8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2172,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005720"},"pg_stats":[{"pgid":"1.f","version":"33'8187","reported_seq":8274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.787417+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:54.787417+0000","last_peered":"2024-09-11T05:01:54.787417+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:54.787417+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8187,"log_dups_size":0,"ondisk_log_size":8187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35446784,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8261","reported_seq":8364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.011061+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:53.011061+0000","last_peered":"2024-09-11T05:01:53.011061+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:53.011061+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672672,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7939","reported_seq":8037,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.404089+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:55.404089+0000","last_peered":"2024-09-11T05:01:55.404089+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:55.404089+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34577000,"num_objects":4222,"num_object_clones":0,"num_object_copies":16888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7938,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8057","reported_seq":8154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.129679+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:53.129679+0000","last_peered":"2024-09-11T05:01:53.129679+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:53.129679+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35706016,"num_objects":4359,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8055,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2095}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8127","reported_seq":8240,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.856789+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:54.856789+0000","last_peered":"2024-09-11T05:01:54.856789+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:54.856789+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074656,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8261","reported_seq":8355,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.842296+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:54.842296+0000","last_peered":"2024-09-11T05:01:54.842296+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:54.842296+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35872768,"num_objects":4379,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8261,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8232","reported_seq":8313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.134392+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:53.134392+0000","last_peered":"2024-09-11T05:01:53.134392+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:53.134392+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36844704,"num_objects":4498,"num_object_clones":0,"num_object_copies":17992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8361","reported_seq":11472,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.149575+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:53.149575+0000","last_peered":"2024-09-11T05:01:53.149575+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:53.149575+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36566176,"num_objects":4464,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8416","reported_seq":8534,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.771802+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:54.771802+0000","last_peered":"2024-09-11T05:01:54.771802+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:54.771802+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35809600,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8222","reported_seq":8329,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.353594+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:55.353594+0000","last_peered":"2024-09-11T05:01:55.353594+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:55.353594+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271840,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8220,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2000}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8241","reported_seq":8347,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.737044+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:54.737044+0000","last_peered":"2024-09-11T05:01:54.737044+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:54.737044+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36138736,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8238,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8016","reported_seq":12612,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:54.151532+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:54.151532+0000","last_peered":"2024-09-11T05:01:54.151532+0000","last_clean":"2024-09-11T05:01:54.151532+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:54.151532+0000","last_undegraded":"2024-09-11T05:01:54.151532+0000","last_fullsized":"2024-09-11T05:01:54.151532+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1826,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8318","reported_seq":8410,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.545178+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:55.545178+0000","last_peered":"2024-09-11T05:01:55.545178+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:55.545178+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8318,"log_dups_size":0,"ondisk_log_size":8318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36084304,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8317,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8221","reported_seq":8318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.558464+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:54.558464+0000","last_peered":"2024-09-11T05:01:54.558464+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:54.558464+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804320,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8136","reported_seq":8248,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.573571+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:55.573571+0000","last_peered":"2024-09-11T05:01:55.573571+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:55.573571+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001504,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1969}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8295","reported_seq":8392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.219287+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:53.219287+0000","last_peered":"2024-09-11T05:01:53.219287+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:53.219287+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8295,"log_dups_size":0,"ondisk_log_size":8295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36713632,"num_objects":4482,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8293,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":574367368,"num_objects":70119,"num_object_clones":0,"num_object_copies":280476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125074,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1782,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1124925440,"data_stored":1124925440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131290,"ondisk_log_size":131290,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749604,"kb_used_data":280592,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93622236,"statfs":{"total":96636764160,"available":95869169664,"internally_reserved":0,"allocated":287326208,"data_stored":287146239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,3,0,4,1,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744488,"kb_used_data":280152,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93627352,"statfs":{"total":96636764160,"available":95874408448,"internally_reserved":0,"allocated":286875648,"data_stored":286695679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,9,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564484,"kb_used_data":257968,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93807356,"statfs":{"total":96636764160,"available":96058732544,"internally_reserved":0,"allocated":264159232,"data_stored":263979263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313847506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747000,"kb_used_data":282216,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93624840,"statfs":{"total":96636764160,"available":95871836160,"internally_reserved":0,"allocated":288989184,"data_stored":288809215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,1,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288260096,"data_stored":288260096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263835648,"data_stored":263835648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286040064,"data_stored":286040064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286789632,"data_stored":286789632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:58.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:01:58.759 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:58.759 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:01:58.784 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-11T05:01:57.759674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":574367368,"num_objects":70119,"num_object_clones":0,"num_object_copies":280476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125074,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1782,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131290,"ondisk_log_size":131290,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805576,"kb_used_data":1100928,"kb_used_omap":14,"kb_used_meta":1704561,"kb_avail":374681784,"statfs":{"total":386547056640,"available":383674146816,"internally_reserved":0,"allocated":1127350272,"data_stored":1126630396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1745471030},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,7,7,17,4,0,0,0,0,1,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17793024,"num_objects":-2172,"num_object_clones":0,"num_object_copies":-8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2172,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005720"},"pg_stats":[{"pgid":"1.f","version":"33'8187","reported_seq":8274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.787417+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:54.787417+0000","last_peered":"2024-09-11T05:01:54.787417+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:54.787417+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8187,"log_dups_size":0,"ondisk_log_size":8187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35446784,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8261","reported_seq":8364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.011061+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:53.011061+0000","last_peered":"2024-09-11T05:01:53.011061+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:53.011061+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672672,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7939","reported_seq":8037,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.404089+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:55.404089+0000","last_peered":"2024-09-11T05:01:55.404089+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:55.404089+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34577000,"num_objects":4222,"num_object_clones":0,"num_object_copies":16888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7938,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8057","reported_seq":8154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.129679+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:53.129679+0000","last_peered":"2024-09-11T05:01:53.129679+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:53.129679+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35706016,"num_objects":4359,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8055,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2095}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8127","reported_seq":8240,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.856789+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:54.856789+0000","last_peered":"2024-09-11T05:01:54.856789+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:54.856789+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074656,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8261","reported_seq":8355,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.842296+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:54.842296+0000","last_peered":"2024-09-11T05:01:54.842296+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:54.842296+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35872768,"num_objects":4379,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8261,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8232","reported_seq":8313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.134392+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:53.134392+0000","last_peered":"2024-09-11T05:01:53.134392+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:53.134392+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36844704,"num_objects":4498,"num_object_clones":0,"num_object_copies":17992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8361","reported_seq":11472,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.149575+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:53.149575+0000","last_peered":"2024-09-11T05:01:53.149575+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:53.149575+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36566176,"num_objects":4464,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8416","reported_seq":8534,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.771802+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:54.771802+0000","last_peered":"2024-09-11T05:01:54.771802+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:54.771802+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35809600,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8222","reported_seq":8329,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.353594+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:55.353594+0000","last_peered":"2024-09-11T05:01:55.353594+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:55.353594+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271840,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8220,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2000}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8241","reported_seq":8347,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.737044+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:54.737044+0000","last_peered":"2024-09-11T05:01:54.737044+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:54.737044+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36138736,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8238,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8016","reported_seq":12612,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:54.151532+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:54.151532+0000","last_peered":"2024-09-11T05:01:54.151532+0000","last_clean":"2024-09-11T05:01:54.151532+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:54.151532+0000","last_undegraded":"2024-09-11T05:01:54.151532+0000","last_fullsized":"2024-09-11T05:01:54.151532+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1826,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8318","reported_seq":8410,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.545178+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:55.545178+0000","last_peered":"2024-09-11T05:01:55.545178+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:55.545178+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8318,"log_dups_size":0,"ondisk_log_size":8318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36084304,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8317,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8221","reported_seq":8318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.558464+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:54.558464+0000","last_peered":"2024-09-11T05:01:54.558464+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:54.558464+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804320,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8136","reported_seq":8248,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.573571+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:55.573571+0000","last_peered":"2024-09-11T05:01:55.573571+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:55.573571+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001504,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1969}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8295","reported_seq":8392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:53.219287+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:53.219287+0000","last_peered":"2024-09-11T05:01:53.219287+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:53.219287+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8295,"log_dups_size":0,"ondisk_log_size":8295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36713632,"num_objects":4482,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8293,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":574367368,"num_objects":70119,"num_object_clones":0,"num_object_copies":280476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125074,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1782,"num_bytes_recovered":12402688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1124925440,"data_stored":1124925440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131290,"ondisk_log_size":131290,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749604,"kb_used_data":280592,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93622236,"statfs":{"total":96636764160,"available":95869169664,"internally_reserved":0,"allocated":287326208,"data_stored":287146239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,3,0,4,1,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744488,"kb_used_data":280152,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93627352,"statfs":{"total":96636764160,"available":95874408448,"internally_reserved":0,"allocated":286875648,"data_stored":286695679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,9,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564484,"kb_used_data":257968,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93807356,"statfs":{"total":96636764160,"available":96058732544,"internally_reserved":0,"allocated":264159232,"data_stored":263979263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":313847506},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747000,"kb_used_data":282216,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93624840,"statfs":{"total":96636764160,"available":95871836160,"internally_reserved":0,"allocated":288989184,"data_stored":288809215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,1,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288260096,"data_stored":288260096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263835648,"data_stored":263835648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286040064,"data_stored":286040064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286789632,"data_stored":286789632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:01:58.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:01:58.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8187", 'reported_seq': 8274, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:54.787417+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:54.787417+0000', 'last_peered': '2024-09-11T05:01:54.787417+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:54.787417+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8187, 'log_dups_size': 0, 'ondisk_log_size': 8187, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35446784, 'num_objects': 4327, 'num_object_clones': 0, 'num_object_copies': 17308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8187, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2035}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:58.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:01:58.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8261", 'reported_seq': 8364, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:53.011061+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:01:53.011061+0000', 'last_peered': '2024-09-11T05:01:53.011061+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:01:53.011061+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8261, 'log_dups_size': 0, 'ondisk_log_size': 8261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36672672, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 17908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8259, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7939", 'reported_seq': 8037, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:55.404089+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:01:55.404089+0000', 'last_peered': '2024-09-11T05:01:55.404089+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:01:55.404089+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 7939, 'log_dups_size': 0, 'ondisk_log_size': 7939, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34577000, 'num_objects': 4222, 'num_object_clones': 0, 'num_object_copies': 16888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4222, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7938, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2010}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8057", 'reported_seq': 8154, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:53.129679+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:01:53.129679+0000', 'last_peered': '2024-09-11T05:01:53.129679+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:01:53.129679+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8057, 'log_dups_size': 0, 'ondisk_log_size': 8057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35706016, 'num_objects': 4359, 'num_object_clones': 0, 'num_object_copies': 17436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8055, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2095}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8127", 'reported_seq': 8240, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:54.856789+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:54.856789+0000', 'last_peered': '2024-09-11T05:01:54.856789+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:54.856789+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8127, 'log_dups_size': 0, 'ondisk_log_size': 8127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36074656, 'num_objects': 4404, 'num_object_clones': 0, 'num_object_copies': 17616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8125, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2181}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8261", 'reported_seq': 8355, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:54.842296+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:54.842296+0000', 'last_peered': '2024-09-11T05:01:54.842296+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:54.842296+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8261, 'log_dups_size': 0, 'ondisk_log_size': 8261, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35872768, 'num_objects': 4379, 'num_object_clones': 0, 'num_object_copies': 17516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8261, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:01:58.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8232", 'reported_seq': 8313, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:53.134392+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:01:53.134392+0000', 'last_peered': '2024-09-11T05:01:53.134392+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:01:53.134392+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8232, 'log_dups_size': 0, 'ondisk_log_size': 8232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36844704, 'num_objects': 4498, 'num_object_clones': 0, 'num_object_copies': 17992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8230, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2187}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8361", 'reported_seq': 11472, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:53.149575+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:01:53.149575+0000', 'last_peered': '2024-09-11T05:01:53.149575+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:01:53.149575+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8361, 'log_dups_size': 0, 'ondisk_log_size': 8361, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36566176, 'num_objects': 4464, 'num_object_clones': 0, 'num_object_copies': 17856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 842, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8359, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1523, 'num_bytes_recovered': 12402688, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8416", 'reported_seq': 8534, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:54.771802+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:54.771802+0000', 'last_peered': '2024-09-11T05:01:54.771802+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:54.771802+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8416, 'log_dups_size': 0, 'ondisk_log_size': 8416, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35809600, 'num_objects': 4372, 'num_object_clones': 0, 'num_object_copies': 17488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8413, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2006}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8222", 'reported_seq': 8329, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:55.353594+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:01:55.353594+0000', 'last_peered': '2024-09-11T05:01:55.353594+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:01:55.353594+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8222, 'log_dups_size': 0, 'ondisk_log_size': 8222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35271840, 'num_objects': 4306, 'num_object_clones': 0, 'num_object_copies': 17224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8220, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2000}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8241", 'reported_seq': 8347, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:54.737044+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:54.737044+0000', 'last_peered': '2024-09-11T05:01:54.737044+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:54.737044+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8241, 'log_dups_size': 0, 'ondisk_log_size': 8241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36138736, 'num_objects': 4412, 'num_object_clones': 0, 'num_object_copies': 17648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8238, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2111}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:01:58.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8318", 'reported_seq': 8410, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:55.545178+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:01:55.545178+0000', 'last_peered': '2024-09-11T05:01:55.545178+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:01:55.545178+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8318, 'log_dups_size': 0, 'ondisk_log_size': 8318, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36084304, 'num_objects': 4405, 'num_object_clones': 0, 'num_object_copies': 17620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8317, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:58.788 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:01:58.788 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8221", 'reported_seq': 8318, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:54.558464+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:54.558464+0000', 'last_peered': '2024-09-11T05:01:54.558464+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:54.558464+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8221, 'log_dups_size': 0, 'ondisk_log_size': 8221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35804320, 'num_objects': 4371, 'num_object_clones': 0, 'num_object_copies': 17484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8219, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:01:58.788 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:01:58.788 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8136", 'reported_seq': 8248, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:55.573571+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:01:55.573571+0000', 'last_peered': '2024-09-11T05:01:55.573571+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:01:55.573571+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8136, 'log_dups_size': 0, 'ondisk_log_size': 8136, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35001504, 'num_objects': 4273, 'num_object_clones': 0, 'num_object_copies': 17092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8134, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1969}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:01:58.788 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:01:58.788 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8295", 'reported_seq': 8392, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:53.219287+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:01:53.219287+0000', 'last_peered': '2024-09-11T05:01:53.219287+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:01:53.219287+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8295, 'log_dups_size': 0, 'ondisk_log_size': 8295, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36713632, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 17928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8293, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:01:58.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:01:59.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:01:59.564 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32891/280476 objects degraded (11.727%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":446,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70119,"data_bytes":574367368,"bytes_used":2872909824,"bytes_avail":383674146816,"bytes_total":386547056640,"degraded_objects":32891,"degraded_total":280476,"degraded_ratio":0.11726850069168128,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":100304,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":1204224,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:01:59.594 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:01:59.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:00.201 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:00.219 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-11T05:01:59.763900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570762888,"num_objects":69679,"num_object_clones":0,"num_object_copies":278716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125514,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1847,"num_bytes_recovered":12926976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131730,"ondisk_log_size":131730,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2827280,"kb_used_data":1096268,"kb_used_omap":14,"kb_used_meta":1730929,"kb_avail":374660080,"statfs":{"total":386547056640,"available":383651921920,"internally_reserved":0,"allocated":1122578432,"data_stored":1121858556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1772471862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,9,3,15,7,25,8,3,0,0,0,1,4,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21397504,"num_objects":-2612,"num_object_clones":0,"num_object_copies":-10448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2612,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008981"},"pg_stats":[{"pgid":"1.f","version":"33'8187","reported_seq":8274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.787417+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:54.787417+0000","last_peered":"2024-09-11T05:01:54.787417+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:54.787417+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8187,"log_dups_size":0,"ondisk_log_size":8187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35446784,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8322","reported_seq":8425,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:58.176642+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:01:58.176642+0000","last_peered":"2024-09-11T05:01:58.176642+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:01:58.176642+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8322,"log_dups_size":0,"ondisk_log_size":8322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36172960,"num_objects":4416,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8320,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7939","reported_seq":8037,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.404089+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:01:55.404089+0000","last_peered":"2024-09-11T05:01:55.404089+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:01:55.404089+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34577000,"num_objects":4222,"num_object_clones":0,"num_object_copies":16888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7938,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8139","reported_seq":8236,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:58.193178+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:01:58.193178+0000","last_peered":"2024-09-11T05:01:58.193178+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:01:58.193178+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8139,"log_dups_size":0,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35034272,"num_objects":4277,"num_object_clones":0,"num_object_copies":17108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8127","reported_seq":8240,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.856789+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:54.856789+0000","last_peered":"2024-09-11T05:01:54.856789+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:54.856789+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074656,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8261","reported_seq":8355,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.842296+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:54.842296+0000","last_peered":"2024-09-11T05:01:54.842296+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:54.842296+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35872768,"num_objects":4379,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8261,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8317","reported_seq":8399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:58.267720+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:01:58.267720+0000","last_peered":"2024-09-11T05:01:58.267720+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:01:58.267720+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8317,"log_dups_size":0,"ondisk_log_size":8317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36148384,"num_objects":4413,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8315,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2103}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8425","reported_seq":11664,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:58.283947+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:01:58.283947+0000","last_peered":"2024-09-11T05:01:58.283947+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:01:58.283947+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36041888,"num_objects":4400,"num_object_clones":0,"num_object_copies":17600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1587,"num_bytes_recovered":12926976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8416","reported_seq":8534,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.771802+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:54.771802+0000","last_peered":"2024-09-11T05:01:54.771802+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:54.771802+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35809600,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8222","reported_seq":8329,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.353594+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:01:55.353594+0000","last_peered":"2024-09-11T05:01:55.353594+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:01:55.353594+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271840,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8220,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2000}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8241","reported_seq":8347,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.737044+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:54.737044+0000","last_peered":"2024-09-11T05:01:54.737044+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:54.737044+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36138736,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8238,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8092","reported_seq":12688,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:59.093233+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:59.093233+0000","last_peered":"2024-09-11T05:01:59.093233+0000","last_clean":"2024-09-11T05:01:59.093233+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:59.093233+0000","last_undegraded":"2024-09-11T05:01:59.093233+0000","last_fullsized":"2024-09-11T05:01:59.093233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35160064,"num_objects":4292,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1902,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8318","reported_seq":8410,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.545178+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:01:55.545178+0000","last_peered":"2024-09-11T05:01:55.545178+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:01:55.545178+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8318,"log_dups_size":0,"ondisk_log_size":8318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36084304,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8317,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8221","reported_seq":8318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:54.558464+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:54.558464+0000","last_peered":"2024-09-11T05:01:54.558464+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:54.558464+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804320,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8136","reported_seq":8248,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:55.573571+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:01:55.573571+0000","last_peered":"2024-09-11T05:01:55.573571+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:01:55.573571+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001504,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1969}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8367","reported_seq":8465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:58.101351+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:01:58.101351+0000","last_peered":"2024-09-11T05:01:58.101351+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:01:58.101351+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8367,"log_dups_size":0,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36123808,"num_objects":4410,"num_object_clones":0,"num_object_copies":17640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":570762888,"num_objects":69679,"num_object_clones":0,"num_object_copies":278716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125514,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1847,"num_bytes_recovered":12926976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1120022528,"data_stored":1120022528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131730,"ondisk_log_size":131730,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749604,"kb_used_data":280592,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93622236,"statfs":{"total":96636764160,"available":95869169664,"internally_reserved":0,"allocated":287326208,"data_stored":287146239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,3,0,4,1,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744488,"kb_used_data":280152,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93627352,"statfs":{"total":96636764160,"available":95874408448,"internally_reserved":0,"allocated":286875648,"data_stored":286695679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,9,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586380,"kb_used_data":257660,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93785460,"statfs":{"total":96636764160,"available":96036311040,"internally_reserved":0,"allocated":263843840,"data_stored":263663871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336588498},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746808,"kb_used_data":277864,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93625032,"statfs":{"total":96636764160,"available":95872032768,"internally_reserved":0,"allocated":284532736,"data_stored":284352767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,0,3,3,8,2,3,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283660288,"data_stored":283660288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263532544,"data_stored":263532544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286040064,"data_stored":286040064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286789632,"data_stored":286789632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:03.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:03.821 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:03.846 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-11T05:02:03.765746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562316936,"num_objects":68648,"num_object_clones":0,"num_object_copies":274592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68648,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126545,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132761,"ondisk_log_size":132761,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2826620,"kb_used_data":1083116,"kb_used_omap":14,"kb_used_meta":1743409,"kb_avail":374660740,"statfs":{"total":386547056640,"available":383652597760,"internally_reserved":0,"allocated":1109110784,"data_stored":1108390908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1785251382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,7,7,12,7,1,0,0,0,2,0,4,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20733952,"num_objects":-2531,"num_object_clones":0,"num_object_copies":-10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1761280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008961"},"pg_stats":[{"pgid":"1.f","version":"33'8251","reported_seq":8338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.546427+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:59.546427+0000","last_peered":"2024-09-11T05:01:59.546427+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:59.546427+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8251,"log_dups_size":0,"ondisk_log_size":8251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34922496,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8251,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8405","reported_seq":8508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.240617+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:03.240617+0000","last_peered":"2024-09-11T05:02:03.240617+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:03.240617+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35493024,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8002","reported_seq":8100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:00.646358+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:00.646358+0000","last_peered":"2024-09-11T05:02:00.646358+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:00.646358+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060904,"num_objects":4159,"num_object_clones":0,"num_object_copies":16636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8001,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1947}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8221","reported_seq":8318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.290867+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:03.290867+0000","last_peered":"2024-09-11T05:02:03.290867+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:03.290867+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34362528,"num_objects":4195,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8189","reported_seq":8305,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.846030+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:59.846030+0000","last_peered":"2024-09-11T05:01:59.846030+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:59.846030+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35566752,"num_objects":4342,"num_object_clones":0,"num_object_copies":17368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8337","reported_seq":8433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.795893+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:59.795893+0000","last_peered":"2024-09-11T05:01:59.795893+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:59.795893+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8337,"log_dups_size":0,"ondisk_log_size":8337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35250176,"num_objects":4303,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8337,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2007}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8384","reported_seq":8467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:02.929912+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:02.929912+0000","last_peered":"2024-09-11T05:02:02.929912+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:02.929912+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599520,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8382,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2036}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8495","reported_seq":11870,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.271357+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:03.271357+0000","last_peered":"2024-09-11T05:02:03.271357+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:03.271357+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8495,"log_dups_size":0,"ondisk_log_size":8495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35468448,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8493,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8473","reported_seq":8591,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.462814+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:59.462814+0000","last_peered":"2024-09-11T05:01:59.462814+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:59.462814+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8473,"log_dups_size":0,"ondisk_log_size":8473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35342656,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8470,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8289","reported_seq":8396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:00.648644+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:00.648644+0000","last_peered":"2024-09-11T05:02:00.648644+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:00.648644+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8289,"log_dups_size":0,"ondisk_log_size":8289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34722976,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8287,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8322","reported_seq":8428,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.739372+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:59.739372+0000","last_peered":"2024-09-11T05:01:59.739372+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:59.739372+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8322,"log_dups_size":0,"ondisk_log_size":8322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35475184,"num_objects":4331,"num_object_clones":0,"num_object_copies":17324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8319,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8092","reported_seq":12688,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:59.093233+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:59.093233+0000","last_peered":"2024-09-11T05:01:59.093233+0000","last_clean":"2024-09-11T05:01:59.093233+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:59.093233+0000","last_undegraded":"2024-09-11T05:01:59.093233+0000","last_fullsized":"2024-09-11T05:01:59.093233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35160064,"num_objects":4292,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1902,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8364","reported_seq":8455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:00.552637+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:00.552637+0000","last_peered":"2024-09-11T05:02:00.552637+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:00.552637+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35707472,"num_objects":4359,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8288","reported_seq":8385,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.698085+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:59.698085+0000","last_peered":"2024-09-11T05:01:59.698085+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:59.698085+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8288,"log_dups_size":0,"ondisk_log_size":8288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35255456,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8286,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8209","reported_seq":8321,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:00.536539+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:00.536539+0000","last_peered":"2024-09-11T05:02:00.536539+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:00.536539+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34403488,"num_objects":4200,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8440","reported_seq":8538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.135116+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:03.135116+0000","last_peered":"2024-09-11T05:02:03.135116+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:03.135116+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35525792,"num_objects":4337,"num_object_clones":0,"num_object_copies":17348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":562316936,"num_objects":68648,"num_object_clones":0,"num_object_copies":274592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68648,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126545,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1106583552,"data_stored":1106583552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132761,"ondisk_log_size":132761,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749296,"kb_used_data":276124,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93622544,"statfs":{"total":96636764160,"available":95869485056,"internally_reserved":0,"allocated":282750976,"data_stored":282571007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,2,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744316,"kb_used_data":275812,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93627524,"statfs":{"total":96636764160,"available":95874584576,"internally_reserved":0,"allocated":282431488,"data_stored":282251519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,6,1,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586380,"kb_used_data":257660,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93785460,"statfs":{"total":96636764160,"available":96036311040,"internally_reserved":0,"allocated":263843840,"data_stored":263663871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336588498},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746628,"kb_used_data":273520,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93625212,"statfs":{"total":96636764160,"available":95872217088,"internally_reserved":0,"allocated":280084480,"data_stored":279904511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,1,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279146496,"data_stored":279146496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263532544,"data_stored":263532544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":281604096,"data_stored":281604096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":282300416,"data_stored":282300416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:03.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:04.478 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:04.499 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-11T05:02:03.765746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562316936,"num_objects":68648,"num_object_clones":0,"num_object_copies":274592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68648,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126545,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132761,"ondisk_log_size":132761,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2826620,"kb_used_data":1083116,"kb_used_omap":14,"kb_used_meta":1743409,"kb_avail":374660740,"statfs":{"total":386547056640,"available":383652597760,"internally_reserved":0,"allocated":1109110784,"data_stored":1108390908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1785251382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,7,7,12,7,1,0,0,0,2,0,4,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20733952,"num_objects":-2531,"num_object_clones":0,"num_object_copies":-10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1761280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008961"},"pg_stats":[{"pgid":"1.f","version":"33'8251","reported_seq":8338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.546427+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:01:59.546427+0000","last_peered":"2024-09-11T05:01:59.546427+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:01:59.546427+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8251,"log_dups_size":0,"ondisk_log_size":8251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34922496,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8251,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8405","reported_seq":8508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.240617+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:03.240617+0000","last_peered":"2024-09-11T05:02:03.240617+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:03.240617+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35493024,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8002","reported_seq":8100,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:00.646358+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:00.646358+0000","last_peered":"2024-09-11T05:02:00.646358+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:00.646358+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34060904,"num_objects":4159,"num_object_clones":0,"num_object_copies":16636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8001,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1947}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8221","reported_seq":8318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.290867+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:03.290867+0000","last_peered":"2024-09-11T05:02:03.290867+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:03.290867+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34362528,"num_objects":4195,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8189","reported_seq":8305,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.846030+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:01:59.846030+0000","last_peered":"2024-09-11T05:01:59.846030+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:01:59.846030+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35566752,"num_objects":4342,"num_object_clones":0,"num_object_copies":17368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8337","reported_seq":8433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.795893+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:01:59.795893+0000","last_peered":"2024-09-11T05:01:59.795893+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:01:59.795893+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8337,"log_dups_size":0,"ondisk_log_size":8337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35250176,"num_objects":4303,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8337,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2007}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8384","reported_seq":8467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:02.929912+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:02.929912+0000","last_peered":"2024-09-11T05:02:02.929912+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:02.929912+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599520,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8382,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2036}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8495","reported_seq":11870,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.271357+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:03.271357+0000","last_peered":"2024-09-11T05:02:03.271357+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:03.271357+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8495,"log_dups_size":0,"ondisk_log_size":8495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35468448,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8493,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8473","reported_seq":8591,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.462814+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:01:59.462814+0000","last_peered":"2024-09-11T05:01:59.462814+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:01:59.462814+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8473,"log_dups_size":0,"ondisk_log_size":8473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35342656,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8470,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8289","reported_seq":8396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:00.648644+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:00.648644+0000","last_peered":"2024-09-11T05:02:00.648644+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:00.648644+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8289,"log_dups_size":0,"ondisk_log_size":8289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34722976,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8287,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8322","reported_seq":8428,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.739372+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:01:59.739372+0000","last_peered":"2024-09-11T05:01:59.739372+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:01:59.739372+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8322,"log_dups_size":0,"ondisk_log_size":8322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35475184,"num_objects":4331,"num_object_clones":0,"num_object_copies":17324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8319,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8092","reported_seq":12688,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:01:59.093233+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:01:59.093233+0000","last_peered":"2024-09-11T05:01:59.093233+0000","last_clean":"2024-09-11T05:01:59.093233+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:01:59.093233+0000","last_undegraded":"2024-09-11T05:01:59.093233+0000","last_fullsized":"2024-09-11T05:01:59.093233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35160064,"num_objects":4292,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1902,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8364","reported_seq":8455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:00.552637+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:00.552637+0000","last_peered":"2024-09-11T05:02:00.552637+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:00.552637+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35707472,"num_objects":4359,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8288","reported_seq":8385,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:01:59.698085+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:01:59.698085+0000","last_peered":"2024-09-11T05:01:59.698085+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:01:59.698085+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8288,"log_dups_size":0,"ondisk_log_size":8288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35255456,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8286,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8209","reported_seq":8321,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:00.536539+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:00.536539+0000","last_peered":"2024-09-11T05:02:00.536539+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:00.536539+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8209,"log_dups_size":0,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34403488,"num_objects":4200,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8440","reported_seq":8538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.135116+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:03.135116+0000","last_peered":"2024-09-11T05:02:03.135116+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:03.135116+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35525792,"num_objects":4337,"num_object_clones":0,"num_object_copies":17348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":562316936,"num_objects":68648,"num_object_clones":0,"num_object_copies":274592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68648,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126545,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1106583552,"data_stored":1106583552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132761,"ondisk_log_size":132761,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749296,"kb_used_data":276124,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93622544,"statfs":{"total":96636764160,"available":95869485056,"internally_reserved":0,"allocated":282750976,"data_stored":282571007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,2,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744316,"kb_used_data":275812,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93627524,"statfs":{"total":96636764160,"available":95874584576,"internally_reserved":0,"allocated":282431488,"data_stored":282251519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,6,1,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586380,"kb_used_data":257660,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93785460,"statfs":{"total":96636764160,"available":96036311040,"internally_reserved":0,"allocated":263843840,"data_stored":263663871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336588498},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746628,"kb_used_data":273520,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93625212,"statfs":{"total":96636764160,"available":95872217088,"internally_reserved":0,"allocated":280084480,"data_stored":279904511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,1,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279146496,"data_stored":279146496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263532544,"data_stored":263532544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":281604096,"data_stored":281604096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":282300416,"data_stored":282300416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:04.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:04.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8251", 'reported_seq': 8338, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:59.546427+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:01:59.546427+0000', 'last_peered': '2024-09-11T05:01:59.546427+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:01:59.546427+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8251, 'log_dups_size': 0, 'ondisk_log_size': 8251, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34922496, 'num_objects': 4263, 'num_object_clones': 0, 'num_object_copies': 17052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8251, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1971}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:04.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:04.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8405", 'reported_seq': 8508, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:03.240617+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:03.240617+0000', 'last_peered': '2024-09-11T05:02:03.240617+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:03.240617+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8405, 'log_dups_size': 0, 'ondisk_log_size': 8405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35493024, 'num_objects': 4333, 'num_object_clones': 0, 'num_object_copies': 17332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8403, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2072}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:04.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:04.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8002", 'reported_seq': 8100, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:00.646358+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:00.646358+0000', 'last_peered': '2024-09-11T05:02:00.646358+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:00.646358+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8002, 'log_dups_size': 0, 'ondisk_log_size': 8002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34060904, 'num_objects': 4159, 'num_object_clones': 0, 'num_object_copies': 16636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4159, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8001, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1947}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:04.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:04.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8221", 'reported_seq': 8318, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:03.290867+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:03.290867+0000', 'last_peered': '2024-09-11T05:02:03.290867+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:03.290867+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8221, 'log_dups_size': 0, 'ondisk_log_size': 8221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34362528, 'num_objects': 4195, 'num_object_clones': 0, 'num_object_copies': 16780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8219, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1931}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:04.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:04.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8189", 'reported_seq': 8305, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:59.846030+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:01:59.846030+0000', 'last_peered': '2024-09-11T05:01:59.846030+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:01:59.846030+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8189, 'log_dups_size': 0, 'ondisk_log_size': 8189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35566752, 'num_objects': 4342, 'num_object_clones': 0, 'num_object_copies': 17368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8187, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2120}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:04.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:04.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8337", 'reported_seq': 8433, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:59.795893+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:01:59.795893+0000', 'last_peered': '2024-09-11T05:01:59.795893+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:01:59.795893+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8337, 'log_dups_size': 0, 'ondisk_log_size': 8337, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35250176, 'num_objects': 4303, 'num_object_clones': 0, 'num_object_copies': 17212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2296, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8337, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2007}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8384", 'reported_seq': 8467, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:02.929912+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:02.929912+0000', 'last_peered': '2024-09-11T05:02:02.929912+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:02.929912+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8384, 'log_dups_size': 0, 'ondisk_log_size': 8384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35599520, 'num_objects': 4346, 'num_object_clones': 0, 'num_object_copies': 17384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8382, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2036}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8495", 'reported_seq': 11870, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:03.271357+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:03.271357+0000', 'last_peered': '2024-09-11T05:02:03.271357+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:03.271357+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8495, 'log_dups_size': 0, 'ondisk_log_size': 8495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35468448, 'num_objects': 4330, 'num_object_clones': 0, 'num_object_copies': 17320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 710, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8493, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1655, 'num_bytes_recovered': 13484032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8473", 'reported_seq': 8591, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:59.462814+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:01:59.462814+0000', 'last_peered': '2024-09-11T05:01:59.462814+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:01:59.462814+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8473, 'log_dups_size': 0, 'ondisk_log_size': 8473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35342656, 'num_objects': 4315, 'num_object_clones': 0, 'num_object_copies': 17260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8470, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1949}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8289", 'reported_seq': 8396, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:00.648644+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:00.648644+0000', 'last_peered': '2024-09-11T05:02:00.648644+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:00.648644+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8289, 'log_dups_size': 0, 'ondisk_log_size': 8289, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34722976, 'num_objects': 4239, 'num_object_clones': 0, 'num_object_copies': 16956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8287, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1933}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:04.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8322", 'reported_seq': 8428, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:59.739372+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:01:59.739372+0000', 'last_peered': '2024-09-11T05:01:59.739372+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:01:59.739372+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8322, 'log_dups_size': 0, 'ondisk_log_size': 8322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35475184, 'num_objects': 4331, 'num_object_clones': 0, 'num_object_copies': 17324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8319, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2030}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:04.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:04.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8364", 'reported_seq': 8455, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:00.552637+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:00.552637+0000', 'last_peered': '2024-09-11T05:02:00.552637+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:00.552637+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8364, 'log_dups_size': 0, 'ondisk_log_size': 8364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35707472, 'num_objects': 4359, 'num_object_clones': 0, 'num_object_copies': 17436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8363, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:04.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:04.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8288", 'reported_seq': 8385, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:01:59.698085+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:01:59.698085+0000', 'last_peered': '2024-09-11T05:01:59.698085+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:01:59.698085+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8288, 'log_dups_size': 0, 'ondisk_log_size': 8288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35255456, 'num_objects': 4304, 'num_object_clones': 0, 'num_object_copies': 17216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8286, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:04.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:04.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8209", 'reported_seq': 8321, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:00.536539+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:00.536539+0000', 'last_peered': '2024-09-11T05:02:00.536539+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:00.536539+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8209, 'log_dups_size': 0, 'ondisk_log_size': 8209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34403488, 'num_objects': 4200, 'num_object_clones': 0, 'num_object_copies': 16800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8207, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1896}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:04.503 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:04.503 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8440", 'reported_seq': 8538, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:03.135116+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:03.135116+0000', 'last_peered': '2024-09-11T05:02:03.135116+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:03.135116+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8440, 'log_dups_size': 0, 'ondisk_log_size': 8440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35525792, 'num_objects': 4337, 'num_object_clones': 0, 'num_object_copies': 17348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8438, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2041}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:04.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:05.237 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32756/274592 objects degraded (11.929%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":452,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68648,"data_bytes":562316936,"bytes_used":2894458880,"bytes_avail":383652597760,"bytes_total":386547056640,"degraded_objects":32756,"degraded_total":274592,"degraded_ratio":0.11928970982402984,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":146663,"recovering_keys_per_sec":0,"num_objects_recovered":226,"num_bytes_recovered":1761280,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-11T04:57:17.635950+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:05.260 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:05.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:05.880 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:05.881 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:05.909 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-11T05:02:05.766674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":556443272,"num_objects":67931,"num_object_clones":0,"num_object_copies":271724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67931,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127262,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1919,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133478,"ondisk_log_size":133478,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2800912,"kb_used_data":1074040,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374686448,"statfs":{"total":386547056640,"available":383678922752,"internally_reserved":0,"allocated":1099816960,"data_stored":1099097084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1768212022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,9,13,12,8,4,0,0,0,0,0,4,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24076288,"num_objects":-2939,"num_object_clones":0,"num_object_copies":-11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2939,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1081344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008904"},"pg_stats":[{"pgid":"1.f","version":"33'8308","reported_seq":8395,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.707526+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:04.707526+0000","last_peered":"2024-09-11T05:02:04.707526+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:04.707526+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34455552,"num_objects":4206,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8405","reported_seq":8508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.240617+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:03.240617+0000","last_peered":"2024-09-11T05:02:03.240617+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:03.240617+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35493024,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8067","reported_seq":8167,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.464004+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:05.464004+0000","last_peered":"2024-09-11T05:02:05.464004+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:05.464004+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528424,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8066,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8221","reported_seq":8318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.290867+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:03.290867+0000","last_peered":"2024-09-11T05:02:03.290867+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:03.290867+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34362528,"num_objects":4195,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8261","reported_seq":8377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.770102+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:04.770102+0000","last_peered":"2024-09-11T05:02:04.770102+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:04.770102+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34976928,"num_objects":4270,"num_object_clones":0,"num_object_copies":17080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8389","reported_seq":8485,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.748850+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:04.748850+0000","last_peered":"2024-09-11T05:02:04.748850+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:04.748850+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34824192,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1955}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8384","reported_seq":8467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:02.929912+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:02.929912+0000","last_peered":"2024-09-11T05:02:02.929912+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:02.929912+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599520,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8382,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2036}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8495","reported_seq":11870,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.271357+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:03.271357+0000","last_peered":"2024-09-11T05:02:03.271357+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:03.271357+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8495,"log_dups_size":0,"ondisk_log_size":8495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35468448,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8493,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8543","reported_seq":8661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.858851+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:04.858851+0000","last_peered":"2024-09-11T05:02:04.858851+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:04.858851+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8543,"log_dups_size":0,"ondisk_log_size":8543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34769216,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8346","reported_seq":8453,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.513897+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:05.513897+0000","last_peered":"2024-09-11T05:02:05.513897+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:05.513897+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256032,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8394","reported_seq":8500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.675750+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:04.675750+0000","last_peered":"2024-09-11T05:02:04.675750+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:04.675750+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34885360,"num_objects":4259,"num_object_clones":0,"num_object_copies":17036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8164","reported_seq":12759,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:04.185372+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:04.185372+0000","last_peered":"2024-09-11T05:02:04.185372+0000","last_clean":"2024-09-11T05:02:04.185372+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:04.185372+0000","last_undegraded":"2024-09-11T05:02:04.185372+0000","last_fullsized":"2024-09-11T05:02:04.185372+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34570240,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1974,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8430","reported_seq":8522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.267605+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:05.267605+0000","last_peered":"2024-09-11T05:02:05.267605+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:05.267605+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35166800,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8429,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1967}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8350","reported_seq":8447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.797086+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:04.797086+0000","last_peered":"2024-09-11T05:02:04.797086+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:04.797086+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34747552,"num_objects":4242,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8348,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8281","reported_seq":8394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.668521+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:05.668521+0000","last_peered":"2024-09-11T05:02:05.668521+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:05.668521+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813664,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8440","reported_seq":8538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.135116+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:03.135116+0000","last_peered":"2024-09-11T05:02:03.135116+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:03.135116+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35525792,"num_objects":4337,"num_object_clones":0,"num_object_copies":17348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":556443272,"num_objects":67931,"num_object_clones":0,"num_object_copies":271724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67931,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127262,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1919,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097437184,"data_stored":1097437184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133478,"ondisk_log_size":133478,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736556,"kb_used_data":271708,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93635284,"statfs":{"total":96636764160,"available":95882530816,"internally_reserved":0,"allocated":278228992,"data_stored":278049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,1,3,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744112,"kb_used_data":271436,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93627728,"statfs":{"total":96636764160,"available":95874793472,"internally_reserved":0,"allocated":277950464,"data_stored":277770495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573616,"kb_used_data":257376,"kb_used_omap":4,"kb_used_meta":316219,"kb_avail":93798224,"statfs":{"total":96636764160,"available":96049381376,"internally_reserved":0,"allocated":263553024,"data_stored":263373055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323808978},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746628,"kb_used_data":273520,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93625212,"statfs":{"total":96636764160,"available":95872217088,"internally_reserved":0,"allocated":280084480,"data_stored":279904511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,1,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279146496,"data_stored":279146496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263245824,"data_stored":263245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":277204992,"data_stored":277204992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277839872,"data_stored":277839872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:08.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:09.504 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:09.505 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:09.538 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-11T05:02:07.767545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":556443272,"num_objects":67931,"num_object_clones":0,"num_object_copies":271724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67931,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127262,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1919,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133478,"ondisk_log_size":133478,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2800912,"kb_used_data":1074040,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374686448,"statfs":{"total":386547056640,"available":383678922752,"internally_reserved":0,"allocated":1099816960,"data_stored":1099097084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1768212022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,9,13,12,8,4,0,0,0,0,0,4,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17924096,"num_objects":-2188,"num_object_clones":0,"num_object_copies":-8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1081344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008904"},"pg_stats":[{"pgid":"1.f","version":"33'8308","reported_seq":8395,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.707526+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:04.707526+0000","last_peered":"2024-09-11T05:02:04.707526+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:04.707526+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34455552,"num_objects":4206,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8405","reported_seq":8508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.240617+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:03.240617+0000","last_peered":"2024-09-11T05:02:03.240617+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:03.240617+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35493024,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8067","reported_seq":8167,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.464004+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:05.464004+0000","last_peered":"2024-09-11T05:02:05.464004+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:05.464004+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528424,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8066,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8221","reported_seq":8318,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.290867+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:03.290867+0000","last_peered":"2024-09-11T05:02:03.290867+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:03.290867+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34362528,"num_objects":4195,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8261","reported_seq":8377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.770102+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:04.770102+0000","last_peered":"2024-09-11T05:02:04.770102+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:04.770102+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34976928,"num_objects":4270,"num_object_clones":0,"num_object_copies":17080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8389","reported_seq":8485,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.748850+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:04.748850+0000","last_peered":"2024-09-11T05:02:04.748850+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:04.748850+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34824192,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1955}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8384","reported_seq":8467,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:02.929912+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:02.929912+0000","last_peered":"2024-09-11T05:02:02.929912+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:02.929912+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599520,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8382,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2036}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8495","reported_seq":11870,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.271357+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:03.271357+0000","last_peered":"2024-09-11T05:02:03.271357+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:03.271357+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8495,"log_dups_size":0,"ondisk_log_size":8495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35468448,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8493,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8543","reported_seq":8661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.858851+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:04.858851+0000","last_peered":"2024-09-11T05:02:04.858851+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:04.858851+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8543,"log_dups_size":0,"ondisk_log_size":8543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34769216,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8346","reported_seq":8453,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.513897+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:05.513897+0000","last_peered":"2024-09-11T05:02:05.513897+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:05.513897+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256032,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8394","reported_seq":8500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.675750+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:04.675750+0000","last_peered":"2024-09-11T05:02:04.675750+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:04.675750+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34885360,"num_objects":4259,"num_object_clones":0,"num_object_copies":17036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8164","reported_seq":12759,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:04.185372+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:04.185372+0000","last_peered":"2024-09-11T05:02:04.185372+0000","last_clean":"2024-09-11T05:02:04.185372+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:04.185372+0000","last_undegraded":"2024-09-11T05:02:04.185372+0000","last_fullsized":"2024-09-11T05:02:04.185372+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34570240,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1974,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8430","reported_seq":8522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.267605+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:05.267605+0000","last_peered":"2024-09-11T05:02:05.267605+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:05.267605+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35166800,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8429,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1967}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8350","reported_seq":8447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.797086+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:04.797086+0000","last_peered":"2024-09-11T05:02:04.797086+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:04.797086+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34747552,"num_objects":4242,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8348,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8281","reported_seq":8394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.668521+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:05.668521+0000","last_peered":"2024-09-11T05:02:05.668521+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:05.668521+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813664,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8440","reported_seq":8538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:03.135116+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:03.135116+0000","last_peered":"2024-09-11T05:02:03.135116+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:03.135116+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35525792,"num_objects":4337,"num_object_clones":0,"num_object_copies":17348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":556443272,"num_objects":67931,"num_object_clones":0,"num_object_copies":271724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67931,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127262,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1919,"num_bytes_recovered":13484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097437184,"data_stored":1097437184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133478,"ondisk_log_size":133478,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736556,"kb_used_data":271708,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93635284,"statfs":{"total":96636764160,"available":95882530816,"internally_reserved":0,"allocated":278228992,"data_stored":278049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,1,3,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744112,"kb_used_data":271436,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93627728,"statfs":{"total":96636764160,"available":95874793472,"internally_reserved":0,"allocated":277950464,"data_stored":277770495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573616,"kb_used_data":257376,"kb_used_omap":4,"kb_used_meta":316219,"kb_avail":93798224,"statfs":{"total":96636764160,"available":96049381376,"internally_reserved":0,"allocated":263553024,"data_stored":263373055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323808978},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746628,"kb_used_data":273520,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93625212,"statfs":{"total":96636764160,"available":95872217088,"internally_reserved":0,"allocated":280084480,"data_stored":279904511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,1,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279146496,"data_stored":279146496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263245824,"data_stored":263245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":277204992,"data_stored":277204992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277839872,"data_stored":277839872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:09.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:10.143 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:10.143 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:10.171 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-11T05:02:09.768436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":553256584,"num_objects":67542,"num_object_clones":0,"num_object_copies":270168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127651,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133867,"ondisk_log_size":133867,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788052,"kb_used_data":1069504,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374699308,"statfs":{"total":386547056640,"available":383692091392,"internally_reserved":0,"allocated":1095172096,"data_stored":1094452220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1759692342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,6,16,7,3,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21110784,"num_objects":-2577,"num_object_clones":0,"num_object_copies":-10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008761"},"pg_stats":[{"pgid":"1.f","version":"33'8308","reported_seq":8395,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.707526+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:04.707526+0000","last_peered":"2024-09-11T05:02:04.707526+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:04.707526+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34455552,"num_objects":4206,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8458","reported_seq":8561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:07.905634+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:07.905634+0000","last_peered":"2024-09-11T05:02:07.905634+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:07.905634+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058848,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8067","reported_seq":8167,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.464004+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:05.464004+0000","last_peered":"2024-09-11T05:02:05.464004+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:05.464004+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528424,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8066,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8285","reported_seq":8382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:08.221326+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:08.221326+0000","last_peered":"2024-09-11T05:02:08.221326+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:08.221326+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8285,"log_dups_size":0,"ondisk_log_size":8285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838240,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8283,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8261","reported_seq":8377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.770102+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:04.770102+0000","last_peered":"2024-09-11T05:02:04.770102+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:04.770102+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34976928,"num_objects":4270,"num_object_clones":0,"num_object_copies":17080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8389","reported_seq":8485,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.748850+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:04.748850+0000","last_peered":"2024-09-11T05:02:04.748850+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:04.748850+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34824192,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1955}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8449","reported_seq":8532,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:08.249784+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:08.249784+0000","last_peered":"2024-09-11T05:02:08.249784+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:08.249784+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8449,"log_dups_size":0,"ondisk_log_size":8449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35067040,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1971}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8567","reported_seq":12066,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:08.339042+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:08.339042+0000","last_peered":"2024-09-11T05:02:08.339042+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:08.339042+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34878624,"num_objects":4258,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8565,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8543","reported_seq":8661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.858851+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:04.858851+0000","last_peered":"2024-09-11T05:02:04.858851+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:04.858851+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8543,"log_dups_size":0,"ondisk_log_size":8543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34769216,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8346","reported_seq":8453,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.513897+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:05.513897+0000","last_peered":"2024-09-11T05:02:05.513897+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:05.513897+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256032,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8394","reported_seq":8500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.675750+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:04.675750+0000","last_peered":"2024-09-11T05:02:04.675750+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:04.675750+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34885360,"num_objects":4259,"num_object_clones":0,"num_object_copies":17036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8227","reported_seq":12823,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:09.122832+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:09.122832+0000","last_peered":"2024-09-11T05:02:09.122832+0000","last_clean":"2024-09-11T05:02:09.122832+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:09.122832+0000","last_undegraded":"2024-09-11T05:02:09.122832+0000","last_fullsized":"2024-09-11T05:02:09.122832+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8227,"log_dups_size":0,"ondisk_log_size":8227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34054144,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2037,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8430","reported_seq":8522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.267605+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:05.267605+0000","last_peered":"2024-09-11T05:02:05.267605+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:05.267605+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35166800,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8429,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1967}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8350","reported_seq":8447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.797086+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:04.797086+0000","last_peered":"2024-09-11T05:02:04.797086+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:04.797086+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34747552,"num_objects":4242,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8348,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8281","reported_seq":8394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.668521+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:05.668521+0000","last_peered":"2024-09-11T05:02:05.668521+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:05.668521+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813664,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8512","reported_seq":8608,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:08.391136+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:08.391136+0000","last_peered":"2024-09-11T05:02:08.391136+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:08.391136+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8512,"log_dups_size":0,"ondisk_log_size":8512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34935968,"num_objects":4265,"num_object_clones":0,"num_object_copies":17060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8510,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":553256584,"num_objects":67542,"num_object_clones":0,"num_object_copies":270168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127651,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1092759552,"data_stored":1092759552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133867,"ondisk_log_size":133867,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736556,"kb_used_data":271708,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93635284,"statfs":{"total":96636764160,"available":95882530816,"internally_reserved":0,"allocated":278228992,"data_stored":278049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,1,3,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744112,"kb_used_data":271436,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93627728,"statfs":{"total":96636764160,"available":95874793472,"internally_reserved":0,"allocated":277950464,"data_stored":277770495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577456,"kb_used_data":257056,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93794384,"statfs":{"total":96636764160,"available":96045449216,"internally_reserved":0,"allocated":263225344,"data_stored":263045375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328068818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729928,"kb_used_data":269304,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93641912,"statfs":{"total":96636764160,"available":95889317888,"internally_reserved":0,"allocated":275767296,"data_stored":275587327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,3,3,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274784256,"data_stored":274784256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262930432,"data_stored":262930432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":277204992,"data_stored":277204992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277839872,"data_stored":277839872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:10.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:10.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8308", 'reported_seq': 8395, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:04.707526+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:04.707526+0000', 'last_peered': '2024-09-11T05:02:04.707526+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:04.707526+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8308, 'log_dups_size': 0, 'ondisk_log_size': 8308, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34455552, 'num_objects': 4206, 'num_object_clones': 0, 'num_object_copies': 16824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8308, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1914}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:10.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:10.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8458", 'reported_seq': 8561, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:07.905634+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:07.905634+0000', 'last_peered': '2024-09-11T05:02:07.905634+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:07.905634+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8458, 'log_dups_size': 0, 'ondisk_log_size': 8458, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35058848, 'num_objects': 4280, 'num_object_clones': 0, 'num_object_copies': 17120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8456, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:10.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:10.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8067", 'reported_seq': 8167, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:05.464004+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:05.464004+0000', 'last_peered': '2024-09-11T05:02:05.464004+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:05.464004+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8067, 'log_dups_size': 0, 'ondisk_log_size': 8067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33528424, 'num_objects': 4094, 'num_object_clones': 0, 'num_object_copies': 16376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4094, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8066, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1883}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8285", 'reported_seq': 8382, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:08.221326+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:08.221326+0000', 'last_peered': '2024-09-11T05:02:08.221326+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:08.221326+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8285, 'log_dups_size': 0, 'ondisk_log_size': 8285, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33838240, 'num_objects': 4131, 'num_object_clones': 0, 'num_object_copies': 16524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8283, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1867}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8261", 'reported_seq': 8377, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:04.770102+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:04.770102+0000', 'last_peered': '2024-09-11T05:02:04.770102+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:04.770102+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8261, 'log_dups_size': 0, 'ondisk_log_size': 8261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34976928, 'num_objects': 4270, 'num_object_clones': 0, 'num_object_copies': 17080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8259, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2048}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8389", 'reported_seq': 8485, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:04.748850+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:04.748850+0000', 'last_peered': '2024-09-11T05:02:04.748850+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:04.748850+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8389, 'log_dups_size': 0, 'ondisk_log_size': 8389, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34824192, 'num_objects': 4251, 'num_object_clones': 0, 'num_object_copies': 17004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2296, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8389, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1955}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8449", 'reported_seq': 8532, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:08.249784+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:08.249784+0000', 'last_peered': '2024-09-11T05:02:08.249784+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:08.249784+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8449, 'log_dups_size': 0, 'ondisk_log_size': 8449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35067040, 'num_objects': 4281, 'num_object_clones': 0, 'num_object_copies': 17124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8447, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1971}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:10.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8567", 'reported_seq': 12066, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:08.339042+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:08.339042+0000', 'last_peered': '2024-09-11T05:02:08.339042+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:08.339042+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8567, 'log_dups_size': 0, 'ondisk_log_size': 8567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34878624, 'num_objects': 4258, 'num_object_clones': 0, 'num_object_copies': 17032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 648, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8565, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1717, 'num_bytes_recovered': 13991936, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3610}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8543", 'reported_seq': 8661, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:04.858851+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:04.858851+0000', 'last_peered': '2024-09-11T05:02:04.858851+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:04.858851+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8543, 'log_dups_size': 0, 'ondisk_log_size': 8543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34769216, 'num_objects': 4245, 'num_object_clones': 0, 'num_object_copies': 16980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8540, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1879}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8346", 'reported_seq': 8453, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:05.513897+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:05.513897+0000', 'last_peered': '2024-09-11T05:02:05.513897+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:05.513897+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8346, 'log_dups_size': 0, 'ondisk_log_size': 8346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34256032, 'num_objects': 4182, 'num_object_clones': 0, 'num_object_copies': 16728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8344, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1876}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8394", 'reported_seq': 8500, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:04.675750+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:04.675750+0000', 'last_peered': '2024-09-11T05:02:04.675750+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:04.675750+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8394, 'log_dups_size': 0, 'ondisk_log_size': 8394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34885360, 'num_objects': 4259, 'num_object_clones': 0, 'num_object_copies': 17036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8391, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1958}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8430", 'reported_seq': 8522, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:05.267605+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:05.267605+0000', 'last_peered': '2024-09-11T05:02:05.267605+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:05.267605+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8430, 'log_dups_size': 0, 'ondisk_log_size': 8430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35166800, 'num_objects': 4293, 'num_object_clones': 0, 'num_object_copies': 17172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8429, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1967}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8350", 'reported_seq': 8447, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:04.797086+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:04.797086+0000', 'last_peered': '2024-09-11T05:02:04.797086+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:04.797086+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8350, 'log_dups_size': 0, 'ondisk_log_size': 8350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34747552, 'num_objects': 4242, 'num_object_clones': 0, 'num_object_copies': 16968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8348, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:10.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:10.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8281", 'reported_seq': 8394, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:05.668521+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:05.668521+0000', 'last_peered': '2024-09-11T05:02:05.668521+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:05.668521+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8281, 'log_dups_size': 0, 'ondisk_log_size': 8281, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33813664, 'num_objects': 4128, 'num_object_clones': 0, 'num_object_copies': 16512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8279, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:10.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:10.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8512", 'reported_seq': 8608, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:08.391136+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:08.391136+0000', 'last_peered': '2024-09-11T05:02:08.391136+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:08.391136+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8512, 'log_dups_size': 0, 'ondisk_log_size': 8512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34935968, 'num_objects': 4265, 'num_object_clones': 0, 'num_object_copies': 17060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8510, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1969}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:10.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:10.800 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:10.800 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32692/270168 objects degraded (12.101%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":457,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67542,"data_bytes":553256584,"bytes_used":2854965248,"bytes_avail":383692091392,"bytes_total":386547056640,"degraded_objects":32692,"degraded_total":270168,"degraded_ratio":0.1210061887418199,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":132340,"recovering_keys_per_sec":0,"num_objects_recovered":199,"num_bytes_recovered":1589248,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":214},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-11T05:02:09.768550+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-11T05:02:10.820 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:10.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:11.475 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:11.475 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:11.501 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-11T05:02:09.768436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":553256584,"num_objects":67542,"num_object_clones":0,"num_object_copies":270168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127651,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133867,"ondisk_log_size":133867,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788052,"kb_used_data":1069504,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374699308,"statfs":{"total":386547056640,"available":383692091392,"internally_reserved":0,"allocated":1095172096,"data_stored":1094452220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1759692342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,6,16,7,3,0,0,0,0,0,0,6],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21110784,"num_objects":-2577,"num_object_clones":0,"num_object_copies":-10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008761"},"pg_stats":[{"pgid":"1.f","version":"33'8308","reported_seq":8395,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.707526+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:04.707526+0000","last_peered":"2024-09-11T05:02:04.707526+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:04.707526+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34455552,"num_objects":4206,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8458","reported_seq":8561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:07.905634+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:07.905634+0000","last_peered":"2024-09-11T05:02:07.905634+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:07.905634+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058848,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8067","reported_seq":8167,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.464004+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:05.464004+0000","last_peered":"2024-09-11T05:02:05.464004+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:05.464004+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528424,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8066,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1883}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8285","reported_seq":8382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:08.221326+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:08.221326+0000","last_peered":"2024-09-11T05:02:08.221326+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:08.221326+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8285,"log_dups_size":0,"ondisk_log_size":8285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838240,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8283,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1867}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8261","reported_seq":8377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.770102+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:04.770102+0000","last_peered":"2024-09-11T05:02:04.770102+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:04.770102+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34976928,"num_objects":4270,"num_object_clones":0,"num_object_copies":17080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8389","reported_seq":8485,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.748850+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:04.748850+0000","last_peered":"2024-09-11T05:02:04.748850+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:04.748850+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34824192,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1955}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8449","reported_seq":8532,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:08.249784+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:08.249784+0000","last_peered":"2024-09-11T05:02:08.249784+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:08.249784+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8449,"log_dups_size":0,"ondisk_log_size":8449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35067040,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1971}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8567","reported_seq":12066,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:08.339042+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:08.339042+0000","last_peered":"2024-09-11T05:02:08.339042+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:08.339042+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34878624,"num_objects":4258,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8565,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8543","reported_seq":8661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.858851+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:04.858851+0000","last_peered":"2024-09-11T05:02:04.858851+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:04.858851+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8543,"log_dups_size":0,"ondisk_log_size":8543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34769216,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8346","reported_seq":8453,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.513897+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:05.513897+0000","last_peered":"2024-09-11T05:02:05.513897+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:05.513897+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256032,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8394","reported_seq":8500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.675750+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:04.675750+0000","last_peered":"2024-09-11T05:02:04.675750+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:04.675750+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34885360,"num_objects":4259,"num_object_clones":0,"num_object_copies":17036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8227","reported_seq":12823,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:09.122832+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:09.122832+0000","last_peered":"2024-09-11T05:02:09.122832+0000","last_clean":"2024-09-11T05:02:09.122832+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:09.122832+0000","last_undegraded":"2024-09-11T05:02:09.122832+0000","last_fullsized":"2024-09-11T05:02:09.122832+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8227,"log_dups_size":0,"ondisk_log_size":8227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34054144,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2037,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8430","reported_seq":8522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.267605+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:05.267605+0000","last_peered":"2024-09-11T05:02:05.267605+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:05.267605+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35166800,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8429,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1967}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8350","reported_seq":8447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:04.797086+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:04.797086+0000","last_peered":"2024-09-11T05:02:04.797086+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:04.797086+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34747552,"num_objects":4242,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8348,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8281","reported_seq":8394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:05.668521+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:05.668521+0000","last_peered":"2024-09-11T05:02:05.668521+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:05.668521+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813664,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8512","reported_seq":8608,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:08.391136+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:08.391136+0000","last_peered":"2024-09-11T05:02:08.391136+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:08.391136+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8512,"log_dups_size":0,"ondisk_log_size":8512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34935968,"num_objects":4265,"num_object_clones":0,"num_object_copies":17060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8510,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":553256584,"num_objects":67542,"num_object_clones":0,"num_object_copies":270168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127651,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1092759552,"data_stored":1092759552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133867,"ondisk_log_size":133867,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736556,"kb_used_data":271708,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93635284,"statfs":{"total":96636764160,"available":95882530816,"internally_reserved":0,"allocated":278228992,"data_stored":278049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,1,3,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744112,"kb_used_data":271436,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93627728,"statfs":{"total":96636764160,"available":95874793472,"internally_reserved":0,"allocated":277950464,"data_stored":277770495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577456,"kb_used_data":257056,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93794384,"statfs":{"total":96636764160,"available":96045449216,"internally_reserved":0,"allocated":263225344,"data_stored":263045375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328068818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729928,"kb_used_data":269304,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93641912,"statfs":{"total":96636764160,"available":95889317888,"internally_reserved":0,"allocated":275767296,"data_stored":275587327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,3,3,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274784256,"data_stored":274784256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262930432,"data_stored":262930432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":277204992,"data_stored":277204992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277839872,"data_stored":277839872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:14.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:15.179 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:15.205 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-11T05:02:13.770240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":545056392,"num_objects":66541,"num_object_clones":0,"num_object_copies":266164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66541,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128652,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134868,"ondisk_log_size":134868,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775720,"kb_used_data":1057192,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374711640,"statfs":{"total":386547056640,"available":383704719360,"internally_reserved":0,"allocated":1082564608,"data_stored":1081844732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1759692342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,9,7,7,7,2,0,0,0,0,4,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20332544,"num_objects":-2482,"num_object_clones":0,"num_object_copies":-9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005451"},"pg_stats":[{"pgid":"1.f","version":"33'8384","reported_seq":8473,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:09.835423+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:09.835423+0000","last_peered":"2024-09-11T05:02:09.835423+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:09.835423+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33832960,"num_objects":4130,"num_object_clones":0,"num_object_copies":16520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8384,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8516","reported_seq":8619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.375452+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:13.375452+0000","last_peered":"2024-09-11T05:02:13.375452+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:13.375452+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34583712,"num_objects":4222,"num_object_clones":0,"num_object_copies":16888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1961}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8137","reported_seq":8237,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:10.502365+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:10.502365+0000","last_peered":"2024-09-11T05:02:10.502365+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:10.502365+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32954984,"num_objects":4024,"num_object_clones":0,"num_object_copies":16096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8136,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1813}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8344","reported_seq":8443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.123072+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:13.123072+0000","last_peered":"2024-09-11T05:02:13.123072+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:13.123072+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1809}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8329","reported_seq":8447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:09.902845+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:09.902845+0000","last_peered":"2024-09-11T05:02:09.902845+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:09.902845+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8329,"log_dups_size":0,"ondisk_log_size":8329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419872,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8327,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1981}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8460","reported_seq":8556,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:09.766018+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:09.766018+0000","last_peered":"2024-09-11T05:02:09.766018+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:09.766018+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8460,"log_dups_size":0,"ondisk_log_size":8460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34242560,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8509","reported_seq":8592,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.344246+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:13.344246+0000","last_peered":"2024-09-11T05:02:13.344246+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:13.344246+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8509,"log_dups_size":0,"ondisk_log_size":8509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34575520,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8507,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1911}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8644","reported_seq":12301,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.388455+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:13.388455+0000","last_peered":"2024-09-11T05:02:13.388455+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:13.388455+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8644,"log_dups_size":0,"ondisk_log_size":8644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34247840,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8609","reported_seq":8727,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:09.903876+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:09.903876+0000","last_peered":"2024-09-11T05:02:09.903876+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:09.903876+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8609,"log_dups_size":0,"ondisk_log_size":8609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34228544,"num_objects":4179,"num_object_clones":0,"num_object_copies":16716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8606,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8412","reported_seq":8519,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:10.331109+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:10.331109+0000","last_peered":"2024-09-11T05:02:10.331109+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:10.331109+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8412,"log_dups_size":0,"ondisk_log_size":8412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33715360,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8410,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1810}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8457","reported_seq":8563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:09.758806+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:09.758806+0000","last_peered":"2024-09-11T05:02:09.758806+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:09.758806+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34369264,"num_objects":4196,"num_object_clones":0,"num_object_copies":16784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8454,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8227","reported_seq":12823,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:09.122832+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:09.122832+0000","last_peered":"2024-09-11T05:02:09.122832+0000","last_clean":"2024-09-11T05:02:09.122832+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:09.122832+0000","last_undegraded":"2024-09-11T05:02:09.122832+0000","last_fullsized":"2024-09-11T05:02:09.122832+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8227,"log_dups_size":0,"ondisk_log_size":8227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34054144,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2037,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8511","reported_seq":8602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:10.593734+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:10.593734+0000","last_peered":"2024-09-11T05:02:10.593734+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:10.593734+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34503248,"num_objects":4212,"num_object_clones":0,"num_object_copies":16848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8510,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8410","reported_seq":8510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:09.726120+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:09.726120+0000","last_peered":"2024-09-11T05:02:09.726120+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:09.726120+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8410,"log_dups_size":0,"ondisk_log_size":8410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256032,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8408,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1917}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8344","reported_seq":8458,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:10.338845+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:10.338845+0000","last_peered":"2024-09-11T05:02:10.338845+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:10.338845+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33297568,"num_objects":4065,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8575","reported_seq":8680,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.120470+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:13.120470+0000","last_peered":"2024-09-11T05:02:13.120470+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:13.120470+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8575,"log_dups_size":0,"ondisk_log_size":8575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419872,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":545056392,"num_objects":66541,"num_object_clones":0,"num_object_copies":266164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66541,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128652,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1080082432,"data_stored":1080082432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134868,"ondisk_log_size":134868,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736616,"kb_used_data":267608,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93635224,"statfs":{"total":96636764160,"available":95882469376,"internally_reserved":0,"allocated":274030592,"data_stored":273850623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,3,2,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731560,"kb_used_data":267224,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93640280,"statfs":{"total":96636764160,"available":95887646720,"internally_reserved":0,"allocated":273637376,"data_stored":273457407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577456,"kb_used_data":257056,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93794384,"statfs":{"total":96636764160,"available":96045449216,"internally_reserved":0,"allocated":263225344,"data_stored":263045375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328068818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730088,"kb_used_data":265304,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93641752,"statfs":{"total":96636764160,"available":95889154048,"internally_reserved":0,"allocated":271671296,"data_stored":271491327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1,0,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270647296,"data_stored":270647296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262930432,"data_stored":262930432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":273002496,"data_stored":273002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":273502208,"data_stored":273502208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:15.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:15.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T05:02:15.602+0000 7fb84effd640 0 --2- 172.21.15.31:0/1642377677 >> [v2:172.21.15.31:3301/0,v1:172.21.15.31:6790/0] conn(0x7fb85009a2d0 0x7fb850122a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T05:02:15.992 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:15.993 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:16.016 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-11T05:02:15.771012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539428488,"num_objects":65854,"num_object_clones":0,"num_object_copies":263416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129339,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135555,"ondisk_log_size":135555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779732,"kb_used_data":1048712,"kb_used_omap":14,"kb_used_meta":1730929,"kb_avail":374707628,"statfs":{"total":386547056640,"available":383700611072,"internally_reserved":0,"allocated":1073881088,"data_stored":1073161212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1772471862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,7,13,3,2,0,0,0,0,0,6,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22888448,"num_objects":-2794,"num_object_clones":0,"num_object_copies":-11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2794,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005265"},"pg_stats":[{"pgid":"1.f","version":"33'8447","reported_seq":8536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.947760+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:14.947760+0000","last_peered":"2024-09-11T05:02:14.947760+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:14.947760+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8447,"log_dups_size":0,"ondisk_log_size":8447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33316864,"num_objects":4067,"num_object_clones":0,"num_object_copies":16268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8516","reported_seq":8619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.375452+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:13.375452+0000","last_peered":"2024-09-11T05:02:13.375452+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:13.375452+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34583712,"num_objects":4222,"num_object_clones":0,"num_object_copies":16888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1961}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8199","reported_seq":8299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.486160+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:15.486160+0000","last_peered":"2024-09-11T05:02:15.486160+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:15.486160+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8199,"log_dups_size":0,"ondisk_log_size":8199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32447080,"num_objects":3962,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8198,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8344","reported_seq":8443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.123072+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:13.123072+0000","last_peered":"2024-09-11T05:02:13.123072+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:13.123072+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1809}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8392","reported_seq":8510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.759317+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:14.759317+0000","last_peered":"2024-09-11T05:02:14.759317+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:14.759317+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8392,"log_dups_size":0,"ondisk_log_size":8392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33903776,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8390,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8522","reported_seq":8618,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.742211+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:14.742211+0000","last_peered":"2024-09-11T05:02:14.742211+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:14.742211+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8522,"log_dups_size":0,"ondisk_log_size":8522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33734656,"num_objects":4118,"num_object_clones":0,"num_object_copies":16472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8522,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8509","reported_seq":8592,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.344246+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:13.344246+0000","last_peered":"2024-09-11T05:02:13.344246+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:13.344246+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8509,"log_dups_size":0,"ondisk_log_size":8509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34575520,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8507,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1911}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8644","reported_seq":12301,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.388455+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:13.388455+0000","last_peered":"2024-09-11T05:02:13.388455+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:13.388455+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8644,"log_dups_size":0,"ondisk_log_size":8644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34247840,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8664","reported_seq":8782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.859662+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:14.859662+0000","last_peered":"2024-09-11T05:02:14.859662+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:14.859662+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33777984,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8471","reported_seq":8578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.592929+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:15.592929+0000","last_peered":"2024-09-11T05:02:15.592929+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:15.592929+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33232032,"num_objects":4057,"num_object_clones":0,"num_object_copies":16228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8527","reported_seq":8630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.960231+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:14.960231+0000","last_peered":"2024-09-11T05:02:14.960231+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:14.960231+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8527,"log_dups_size":0,"ondisk_log_size":8527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33795824,"num_objects":4126,"num_object_clones":0,"num_object_copies":16504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8298","reported_seq":12894,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:14.016071+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:14.016071+0000","last_peered":"2024-09-11T05:02:14.016071+0000","last_clean":"2024-09-11T05:02:14.016071+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:14.016071+0000","last_undegraded":"2024-09-11T05:02:14.016071+0000","last_fullsized":"2024-09-11T05:02:14.016071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33472512,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8587","reported_seq":8679,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.628888+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:15.628888+0000","last_peered":"2024-09-11T05:02:15.628888+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:15.628888+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33880656,"num_objects":4136,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8586,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1810}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8463","reported_seq":8563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.857954+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:14.857954+0000","last_peered":"2024-09-11T05:02:14.857954+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:14.857954+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8463,"log_dups_size":0,"ondisk_log_size":8463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33821856,"num_objects":4129,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8461,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8397","reported_seq":8511,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.578945+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:15.578945+0000","last_peered":"2024-09-11T05:02:15.578945+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:15.578945+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8397,"log_dups_size":0,"ondisk_log_size":8397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32863392,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8395,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8575","reported_seq":8680,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.120470+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:13.120470+0000","last_peered":"2024-09-11T05:02:13.120470+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:13.120470+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8575,"log_dups_size":0,"ondisk_log_size":8575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419872,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539428488,"num_objects":65854,"num_object_clones":0,"num_object_copies":263416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129339,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1071333376,"data_stored":1071333376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135555,"ondisk_log_size":135555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736648,"kb_used_data":263476,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93635192,"statfs":{"total":96636764160,"available":95882436608,"internally_reserved":0,"allocated":269799424,"data_stored":269619455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,0,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731640,"kb_used_data":263144,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93640200,"statfs":{"total":96636764160,"available":95887564800,"internally_reserved":0,"allocated":269459456,"data_stored":269279487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581356,"kb_used_data":256788,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93790484,"statfs":{"total":96636764160,"available":96041455616,"internally_reserved":0,"allocated":262950912,"data_stored":262770943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332328658},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730088,"kb_used_data":265304,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93641752,"statfs":{"total":96636764160,"available":95889154048,"internally_reserved":0,"allocated":271671296,"data_stored":271491327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1,0,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270647296,"data_stored":270647296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262688768,"data_stored":262688768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268701696,"data_stored":268701696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269295616,"data_stored":269295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:16.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:16.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8447", 'reported_seq': 8536, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:14.947760+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:14.947760+0000', 'last_peered': '2024-09-11T05:02:14.947760+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:14.947760+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8447, 'log_dups_size': 0, 'ondisk_log_size': 8447, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33316864, 'num_objects': 4067, 'num_object_clones': 0, 'num_object_copies': 16268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8447, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1776}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:16.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:16.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8516", 'reported_seq': 8619, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:13.375452+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:13.375452+0000', 'last_peered': '2024-09-11T05:02:13.375452+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:13.375452+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8516, 'log_dups_size': 0, 'ondisk_log_size': 8516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34583712, 'num_objects': 4222, 'num_object_clones': 0, 'num_object_copies': 16888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8514, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1961}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:16.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:16.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8199", 'reported_seq': 8299, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:15.486160+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:15.486160+0000', 'last_peered': '2024-09-11T05:02:15.486160+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:15.486160+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8199, 'log_dups_size': 0, 'ondisk_log_size': 8199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32447080, 'num_objects': 3962, 'num_object_clones': 0, 'num_object_copies': 15848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3962, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8198, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1751}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:16.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8344", 'reported_seq': 8443, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:13.123072+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:13.123072+0000', 'last_peered': '2024-09-11T05:02:13.123072+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:13.123072+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8344, 'log_dups_size': 0, 'ondisk_log_size': 8344, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33354912, 'num_objects': 4072, 'num_object_clones': 0, 'num_object_copies': 16288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8342, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1809}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8392", 'reported_seq': 8510, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:14.759317+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:14.759317+0000', 'last_peered': '2024-09-11T05:02:14.759317+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:14.759317+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8392, 'log_dups_size': 0, 'ondisk_log_size': 8392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33903776, 'num_objects': 4139, 'num_object_clones': 0, 'num_object_copies': 16556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8390, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1918}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8522", 'reported_seq': 8618, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:14.742211+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:14.742211+0000', 'last_peered': '2024-09-11T05:02:14.742211+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:14.742211+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8522, 'log_dups_size': 0, 'ondisk_log_size': 8522, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33734656, 'num_objects': 4118, 'num_object_clones': 0, 'num_object_copies': 16472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2296, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8522, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1822}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8509", 'reported_seq': 8592, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:13.344246+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:13.344246+0000', 'last_peered': '2024-09-11T05:02:13.344246+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:13.344246+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8509, 'log_dups_size': 0, 'ondisk_log_size': 8509, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34575520, 'num_objects': 4221, 'num_object_clones': 0, 'num_object_copies': 16884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8507, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1911}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8644", 'reported_seq': 12301, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:13.388455+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:13.388455+0000', 'last_peered': '2024-09-11T05:02:13.388455+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:13.388455+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8644, 'log_dups_size': 0, 'ondisk_log_size': 8644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34247840, 'num_objects': 4181, 'num_object_clones': 0, 'num_object_copies': 16724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 569, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8642, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1796, 'num_bytes_recovered': 14639104, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:16.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8664", 'reported_seq': 8782, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:14.859662+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:14.859662+0000', 'last_peered': '2024-09-11T05:02:14.859662+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:14.859662+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8664, 'log_dups_size': 0, 'ondisk_log_size': 8664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33777984, 'num_objects': 4124, 'num_object_clones': 0, 'num_object_copies': 16496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8661, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1758}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8471", 'reported_seq': 8578, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:15.592929+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:15.592929+0000', 'last_peered': '2024-09-11T05:02:15.592929+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:15.592929+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8471, 'log_dups_size': 0, 'ondisk_log_size': 8471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33232032, 'num_objects': 4057, 'num_object_clones': 0, 'num_object_copies': 16228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8469, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1751}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8527", 'reported_seq': 8630, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:14.960231+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:14.960231+0000', 'last_peered': '2024-09-11T05:02:14.960231+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:14.960231+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8527, 'log_dups_size': 0, 'ondisk_log_size': 8527, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33795824, 'num_objects': 4126, 'num_object_clones': 0, 'num_object_copies': 16504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8524, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1825}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8587", 'reported_seq': 8679, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:15.628888+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:15.628888+0000', 'last_peered': '2024-09-11T05:02:15.628888+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:15.628888+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8587, 'log_dups_size': 0, 'ondisk_log_size': 8587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33880656, 'num_objects': 4136, 'num_object_clones': 0, 'num_object_copies': 16544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8586, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1810}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8463", 'reported_seq': 8563, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:14.857954+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:14.857954+0000', 'last_peered': '2024-09-11T05:02:14.857954+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:14.857954+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8463, 'log_dups_size': 0, 'ondisk_log_size': 8463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33821856, 'num_objects': 4129, 'num_object_clones': 0, 'num_object_copies': 16516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2267, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8461, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1864}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:16.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8397", 'reported_seq': 8511, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:15.578945+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:15.578945+0000', 'last_peered': '2024-09-11T05:02:15.578945+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:15.578945+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8397, 'log_dups_size': 0, 'ondisk_log_size': 8397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32863392, 'num_objects': 4012, 'num_object_clones': 0, 'num_object_copies': 16048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8395, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1709}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:16.020 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:16.020 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8575", 'reported_seq': 8680, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:13.120470+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:13.120470+0000', 'last_peered': '2024-09-11T05:02:13.120470+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:13.120470+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8575, 'log_dups_size': 0, 'ondisk_log_size': 8575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34419872, 'num_objects': 4202, 'num_object_clones': 0, 'num_object_copies': 16808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8573, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1909}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:16.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:16.844 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32606/263416 objects degraded (12.378%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":463,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65854,"data_bytes":539428488,"bytes_used":2846445568,"bytes_avail":383700611072,"bytes_total":386547056640,"degraded_objects":32606,"degraded_total":263416,"degraded_ratio":0.1237813952075804,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":96213,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":1155072,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":232},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:16.862 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:16.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:17.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:17.455 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:17.479 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-11T05:02:15.771012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539428488,"num_objects":65854,"num_object_clones":0,"num_object_copies":263416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129339,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135555,"ondisk_log_size":135555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779732,"kb_used_data":1048712,"kb_used_omap":14,"kb_used_meta":1730929,"kb_avail":374707628,"statfs":{"total":386547056640,"available":383700611072,"internally_reserved":0,"allocated":1073881088,"data_stored":1073161212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1772471862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,7,13,3,2,0,0,0,0,0,6,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22888448,"num_objects":-2794,"num_object_clones":0,"num_object_copies":-11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2794,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005265"},"pg_stats":[{"pgid":"1.f","version":"33'8447","reported_seq":8536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.947760+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:14.947760+0000","last_peered":"2024-09-11T05:02:14.947760+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:14.947760+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8447,"log_dups_size":0,"ondisk_log_size":8447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33316864,"num_objects":4067,"num_object_clones":0,"num_object_copies":16268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8516","reported_seq":8619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.375452+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:13.375452+0000","last_peered":"2024-09-11T05:02:13.375452+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:13.375452+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34583712,"num_objects":4222,"num_object_clones":0,"num_object_copies":16888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1961}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8199","reported_seq":8299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.486160+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:15.486160+0000","last_peered":"2024-09-11T05:02:15.486160+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:15.486160+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8199,"log_dups_size":0,"ondisk_log_size":8199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32447080,"num_objects":3962,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8198,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8344","reported_seq":8443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.123072+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:13.123072+0000","last_peered":"2024-09-11T05:02:13.123072+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:13.123072+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33354912,"num_objects":4072,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1809}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8392","reported_seq":8510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.759317+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:14.759317+0000","last_peered":"2024-09-11T05:02:14.759317+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:14.759317+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8392,"log_dups_size":0,"ondisk_log_size":8392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33903776,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8390,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8522","reported_seq":8618,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.742211+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:14.742211+0000","last_peered":"2024-09-11T05:02:14.742211+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:14.742211+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8522,"log_dups_size":0,"ondisk_log_size":8522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33734656,"num_objects":4118,"num_object_clones":0,"num_object_copies":16472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8522,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8509","reported_seq":8592,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.344246+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:13.344246+0000","last_peered":"2024-09-11T05:02:13.344246+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:13.344246+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8509,"log_dups_size":0,"ondisk_log_size":8509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34575520,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8507,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1911}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8644","reported_seq":12301,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.388455+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:13.388455+0000","last_peered":"2024-09-11T05:02:13.388455+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:13.388455+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8644,"log_dups_size":0,"ondisk_log_size":8644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34247840,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8664","reported_seq":8782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.859662+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:14.859662+0000","last_peered":"2024-09-11T05:02:14.859662+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:14.859662+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33777984,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8471","reported_seq":8578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.592929+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:15.592929+0000","last_peered":"2024-09-11T05:02:15.592929+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:15.592929+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33232032,"num_objects":4057,"num_object_clones":0,"num_object_copies":16228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8527","reported_seq":8630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.960231+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:14.960231+0000","last_peered":"2024-09-11T05:02:14.960231+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:14.960231+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8527,"log_dups_size":0,"ondisk_log_size":8527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33795824,"num_objects":4126,"num_object_clones":0,"num_object_copies":16504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8298","reported_seq":12894,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:14.016071+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:14.016071+0000","last_peered":"2024-09-11T05:02:14.016071+0000","last_clean":"2024-09-11T05:02:14.016071+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:14.016071+0000","last_undegraded":"2024-09-11T05:02:14.016071+0000","last_fullsized":"2024-09-11T05:02:14.016071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33472512,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8587","reported_seq":8679,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.628888+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:15.628888+0000","last_peered":"2024-09-11T05:02:15.628888+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:15.628888+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33880656,"num_objects":4136,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8586,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1810}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8463","reported_seq":8563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.857954+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:14.857954+0000","last_peered":"2024-09-11T05:02:14.857954+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:14.857954+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8463,"log_dups_size":0,"ondisk_log_size":8463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33821856,"num_objects":4129,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8461,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8397","reported_seq":8511,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.578945+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:15.578945+0000","last_peered":"2024-09-11T05:02:15.578945+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:15.578945+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8397,"log_dups_size":0,"ondisk_log_size":8397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32863392,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8395,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8575","reported_seq":8680,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:13.120470+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:13.120470+0000","last_peered":"2024-09-11T05:02:13.120470+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:13.120470+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8575,"log_dups_size":0,"ondisk_log_size":8575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419872,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539428488,"num_objects":65854,"num_object_clones":0,"num_object_copies":263416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129339,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":14639104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1071333376,"data_stored":1071333376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135555,"ondisk_log_size":135555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736648,"kb_used_data":263476,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93635192,"statfs":{"total":96636764160,"available":95882436608,"internally_reserved":0,"allocated":269799424,"data_stored":269619455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,0,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731640,"kb_used_data":263144,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93640200,"statfs":{"total":96636764160,"available":95887564800,"internally_reserved":0,"allocated":269459456,"data_stored":269279487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581356,"kb_used_data":256788,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93790484,"statfs":{"total":96636764160,"available":96041455616,"internally_reserved":0,"allocated":262950912,"data_stored":262770943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332328658},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730088,"kb_used_data":265304,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93641752,"statfs":{"total":96636764160,"available":95889154048,"internally_reserved":0,"allocated":271671296,"data_stored":271491327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1,0,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270647296,"data_stored":270647296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262688768,"data_stored":262688768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268701696,"data_stored":268701696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269295616,"data_stored":269295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:20.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:21.148 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:21.175 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-11T05:02:19.772604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":536266376,"num_objects":65468,"num_object_clones":0,"num_object_copies":261872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129725,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135941,"ondisk_log_size":135941,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2787984,"kb_used_data":1044492,"kb_used_omap":14,"kb_used_meta":1743409,"kb_avail":374699376,"statfs":{"total":386547056640,"available":383692161024,"internally_reserved":0,"allocated":1069559808,"data_stored":1068839932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1785251382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,15,3,9,0,0,0,0,1,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20176896,"num_objects":-2463,"num_object_clones":0,"num_object_copies":-9852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005059"},"pg_stats":[{"pgid":"1.f","version":"33'8447","reported_seq":8536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.947760+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:14.947760+0000","last_peered":"2024-09-11T05:02:14.947760+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:14.947760+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8447,"log_dups_size":0,"ondisk_log_size":8447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33316864,"num_objects":4067,"num_object_clones":0,"num_object_copies":16268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8580","reported_seq":8688,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.276719+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:18.276719+0000","last_peered":"2024-09-11T05:02:18.276719+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:18.276719+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8580,"log_dups_size":0,"ondisk_log_size":8580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059424,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1899}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8199","reported_seq":8299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.486160+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:15.486160+0000","last_peered":"2024-09-11T05:02:15.486160+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:15.486160+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8199,"log_dups_size":0,"ondisk_log_size":8199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32447080,"num_objects":3962,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8198,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8406","reported_seq":8507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.311656+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:18.311656+0000","last_peered":"2024-09-11T05:02:18.311656+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:18.311656+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847008,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8392","reported_seq":8510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.759317+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:14.759317+0000","last_peered":"2024-09-11T05:02:14.759317+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:14.759317+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8392,"log_dups_size":0,"ondisk_log_size":8392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33903776,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8390,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8522","reported_seq":8618,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.742211+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:14.742211+0000","last_peered":"2024-09-11T05:02:14.742211+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:14.742211+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8522,"log_dups_size":0,"ondisk_log_size":8522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33734656,"num_objects":4118,"num_object_clones":0,"num_object_copies":16472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8522,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8589","reported_seq":8676,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.303605+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:18.303605+0000","last_peered":"2024-09-11T05:02:18.303605+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:18.303605+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33920160,"num_objects":4141,"num_object_clones":0,"num_object_copies":16564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8587,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8695","reported_seq":12508,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.312451+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:18.312451+0000","last_peered":"2024-09-11T05:02:18.312451+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:18.312451+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33830048,"num_objects":4130,"num_object_clones":0,"num_object_copies":16520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1874,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8664","reported_seq":8782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.859662+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:14.859662+0000","last_peered":"2024-09-11T05:02:14.859662+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:14.859662+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33777984,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8471","reported_seq":8578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.592929+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:15.592929+0000","last_peered":"2024-09-11T05:02:15.592929+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:15.592929+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33232032,"num_objects":4057,"num_object_clones":0,"num_object_copies":16228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8527","reported_seq":8630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.960231+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:14.960231+0000","last_peered":"2024-09-11T05:02:14.960231+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:14.960231+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8527,"log_dups_size":0,"ondisk_log_size":8527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33795824,"num_objects":4126,"num_object_clones":0,"num_object_copies":16504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8363","reported_seq":12959,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:19.310286+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:19.310286+0000","last_peered":"2024-09-11T05:02:19.310286+0000","last_clean":"2024-09-11T05:02:19.310286+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:19.310286+0000","last_undegraded":"2024-09-11T05:02:19.310286+0000","last_fullsized":"2024-09-11T05:02:19.310286+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8363,"log_dups_size":0,"ondisk_log_size":8363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32940032,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8587","reported_seq":8679,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.628888+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:15.628888+0000","last_peered":"2024-09-11T05:02:15.628888+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:15.628888+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8587,"log_dups_size":0,"ondisk_log_size":8587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33880656,"num_objects":4136,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8586,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1810}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8463","reported_seq":8563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:14.857954+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:14.857954+0000","last_peered":"2024-09-11T05:02:14.857954+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:14.857954+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8463,"log_dups_size":0,"ondisk_log_size":8463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33821856,"num_objects":4129,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8461,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8397","reported_seq":8511,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:15.578945+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:15.578945+0000","last_peered":"2024-09-11T05:02:15.578945+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:15.578945+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8397,"log_dups_size":0,"ondisk_log_size":8397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32863392,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8395,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8639","reported_seq":8750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.438452+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:18.438452+0000","last_peered":"2024-09-11T05:02:18.438452+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:18.438452+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33895584,"num_objects":4138,"num_object_clones":0,"num_object_copies":16552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8637,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":536266376,"num_objects":65468,"num_object_clones":0,"num_object_copies":261872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129725,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1066856448,"data_stored":1066856448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135941,"ondisk_log_size":135941,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736648,"kb_used_data":263476,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93635192,"statfs":{"total":96636764160,"available":95882436608,"internally_reserved":0,"allocated":269799424,"data_stored":269619455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,0,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731640,"kb_used_data":263144,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93640200,"statfs":{"total":96636764160,"available":95887564800,"internally_reserved":0,"allocated":269459456,"data_stored":269279487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589548,"kb_used_data":256668,"kb_used_omap":4,"kb_used_meta":332859,"kb_avail":93782292,"statfs":{"total":96636764160,"available":96033067008,"internally_reserved":0,"allocated":262828032,"data_stored":262648063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":340848338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730148,"kb_used_data":261204,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93641692,"statfs":{"total":96636764160,"available":95889092608,"internally_reserved":0,"allocated":267472896,"data_stored":267292927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,1,4,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266362880,"data_stored":266362880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262496256,"data_stored":262496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268701696,"data_stored":268701696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269295616,"data_stored":269295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:21.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:21.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:21.814 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:21.844 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-11T05:02:21.773322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530450056,"num_objects":64758,"num_object_clones":0,"num_object_copies":259032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130435,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136651,"ondisk_log_size":136651,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771208,"kb_used_data":1035988,"kb_used_omap":14,"kb_used_meta":1735089,"kb_avail":374716152,"statfs":{"total":386547056640,"available":383709339648,"internally_reserved":0,"allocated":1060851712,"data_stored":1060131836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1776731702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,13,8,14,1,0,0,0,1,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22806528,"num_objects":-2784,"num_object_clones":0,"num_object_copies":-11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004885"},"pg_stats":[{"pgid":"1.f","version":"33'8520","reported_seq":8609,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.010061+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:20.010061+0000","last_peered":"2024-09-11T05:02:20.010061+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:20.010061+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32718848,"num_objects":3994,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1703}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8580","reported_seq":8688,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.276719+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:18.276719+0000","last_peered":"2024-09-11T05:02:18.276719+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:18.276719+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8580,"log_dups_size":0,"ondisk_log_size":8580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059424,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1899}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8265","reported_seq":8363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.609113+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:20.609113+0000","last_peered":"2024-09-11T05:02:20.609113+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:20.609113+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31906408,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8264,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8406","reported_seq":8507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.311656+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:18.311656+0000","last_peered":"2024-09-11T05:02:18.311656+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:18.311656+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847008,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8472","reported_seq":8590,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.927334+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:19.927334+0000","last_peered":"2024-09-11T05:02:19.927334+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:19.927334+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8472,"log_dups_size":0,"ondisk_log_size":8472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33248416,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8470,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8599","reported_seq":8696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.952667+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:19.952667+0000","last_peered":"2024-09-11T05:02:19.952667+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:19.952667+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8599,"log_dups_size":0,"ondisk_log_size":8599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33103872,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8589","reported_seq":8676,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.303605+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:18.303605+0000","last_peered":"2024-09-11T05:02:18.303605+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:18.303605+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33920160,"num_objects":4141,"num_object_clones":0,"num_object_copies":16564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8587,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8695","reported_seq":12508,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.312451+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:18.312451+0000","last_peered":"2024-09-11T05:02:18.312451+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:18.312451+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33830048,"num_objects":4130,"num_object_clones":0,"num_object_copies":16520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1874,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8737","reported_seq":8855,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.779078+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:19.779078+0000","last_peered":"2024-09-11T05:02:19.779078+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:19.779078+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8737,"log_dups_size":0,"ondisk_log_size":8737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33179968,"num_objects":4051,"num_object_clones":0,"num_object_copies":16204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8734,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8529","reported_seq":8636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.714620+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:20.714620+0000","last_peered":"2024-09-11T05:02:20.714620+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:20.714620+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756896,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8527,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8592","reported_seq":8698,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.984787+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:19.984787+0000","last_peered":"2024-09-11T05:02:19.984787+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:19.984787+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33263344,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8363","reported_seq":12959,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:19.310286+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:19.310286+0000","last_peered":"2024-09-11T05:02:19.310286+0000","last_clean":"2024-09-11T05:02:19.310286+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:19.310286+0000","last_undegraded":"2024-09-11T05:02:19.310286+0000","last_fullsized":"2024-09-11T05:02:19.310286+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8363,"log_dups_size":0,"ondisk_log_size":8363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32940032,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8655","reported_seq":8747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.650706+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:20.650706+0000","last_peered":"2024-09-11T05:02:20.650706+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:20.650706+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33323600,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1742}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8530","reported_seq":8630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.866089+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:19.866089+0000","last_peered":"2024-09-11T05:02:19.866089+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:19.866089+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33272992,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1797}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8480","reported_seq":8594,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.761536+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:20.761536+0000","last_peered":"2024-09-11T05:02:20.761536+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:20.761536+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183456,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1626}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8639","reported_seq":8750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.438452+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:18.438452+0000","last_peered":"2024-09-11T05:02:18.438452+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:18.438452+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33895584,"num_objects":4138,"num_object_clones":0,"num_object_copies":16552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8637,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530450056,"num_objects":64758,"num_object_clones":0,"num_object_copies":259032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130435,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1057947648,"data_stored":1057947648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136651,"ondisk_log_size":136651,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719948,"kb_used_data":259240,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93651892,"statfs":{"total":96636764160,"available":95899537408,"internally_reserved":0,"allocated":265461760,"data_stored":265281791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,1,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731564,"kb_used_data":258876,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93640276,"statfs":{"total":96636764160,"available":95887642624,"internally_reserved":0,"allocated":265089024,"data_stored":264909055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589548,"kb_used_data":256668,"kb_used_omap":4,"kb_used_meta":332859,"kb_avail":93782292,"statfs":{"total":96636764160,"available":96033067008,"internally_reserved":0,"allocated":262828032,"data_stored":262648063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":340848338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730148,"kb_used_data":261204,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93641692,"statfs":{"total":96636764160,"available":95889092608,"internally_reserved":0,"allocated":267472896,"data_stored":267292927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,1,4,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266362880,"data_stored":266362880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262496256,"data_stored":262496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264196096,"data_stored":264196096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":264892416,"data_stored":264892416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:21.845 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:21.845 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8520", 'reported_seq': 8609, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.010061+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:20.010061+0000', 'last_peered': '2024-09-11T05:02:20.010061+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:20.010061+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8520, 'log_dups_size': 0, 'ondisk_log_size': 8520, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32718848, 'num_objects': 3994, 'num_object_clones': 0, 'num_object_copies': 15976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8520, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1703}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8580", 'reported_seq': 8688, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:18.276719+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:18.276719+0000', 'last_peered': '2024-09-11T05:02:18.276719+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:18.276719+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8580, 'log_dups_size': 0, 'ondisk_log_size': 8580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34059424, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8578, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1899}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8265", 'reported_seq': 8363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.609113+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:20.609113+0000', 'last_peered': '2024-09-11T05:02:20.609113+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:20.609113+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8265, 'log_dups_size': 0, 'ondisk_log_size': 8265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31906408, 'num_objects': 3896, 'num_object_clones': 0, 'num_object_copies': 15584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3896, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8264, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1685}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8406", 'reported_seq': 8507, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:18.311656+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:18.311656+0000', 'last_peered': '2024-09-11T05:02:18.311656+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:18.311656+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8406, 'log_dups_size': 0, 'ondisk_log_size': 8406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32847008, 'num_objects': 4010, 'num_object_clones': 0, 'num_object_copies': 16040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8404, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8472", 'reported_seq': 8590, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:19.927334+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:19.927334+0000', 'last_peered': '2024-09-11T05:02:19.927334+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:19.927334+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8472, 'log_dups_size': 0, 'ondisk_log_size': 8472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33248416, 'num_objects': 4059, 'num_object_clones': 0, 'num_object_copies': 16236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8470, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1838}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:21.846 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8599", 'reported_seq': 8696, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:19.952667+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:19.952667+0000', 'last_peered': '2024-09-11T05:02:19.952667+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:19.952667+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8599, 'log_dups_size': 0, 'ondisk_log_size': 8599, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33103872, 'num_objects': 4041, 'num_object_clones': 0, 'num_object_copies': 16164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2296, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8599, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1745}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8589", 'reported_seq': 8676, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:18.303605+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:18.303605+0000', 'last_peered': '2024-09-11T05:02:18.303605+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:18.303605+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8589, 'log_dups_size': 0, 'ondisk_log_size': 8589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33920160, 'num_objects': 4141, 'num_object_clones': 0, 'num_object_copies': 16564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8587, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1833}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8695", 'reported_seq': 12508, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:18.312451+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:18.312451+0000', 'last_peered': '2024-09-11T05:02:18.312451+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:18.312451+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8695, 'log_dups_size': 0, 'ondisk_log_size': 8695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33830048, 'num_objects': 4130, 'num_object_clones': 0, 'num_object_copies': 16520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 491, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8693, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1874, 'num_bytes_recovered': 15278080, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8737", 'reported_seq': 8855, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:19.779078+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:19.779078+0000', 'last_peered': '2024-09-11T05:02:19.779078+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:19.779078+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8737, 'log_dups_size': 0, 'ondisk_log_size': 8737, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33179968, 'num_objects': 4051, 'num_object_clones': 0, 'num_object_copies': 16204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8734, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1685}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8529", 'reported_seq': 8636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.714620+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:20.714620+0000', 'last_peered': '2024-09-11T05:02:20.714620+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:20.714620+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8529, 'log_dups_size': 0, 'ondisk_log_size': 8529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32756896, 'num_objects': 3999, 'num_object_clones': 0, 'num_object_copies': 15996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8527, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1693}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8592", 'reported_seq': 8698, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:19.984787+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:19.984787+0000', 'last_peered': '2024-09-11T05:02:19.984787+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:19.984787+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8592, 'log_dups_size': 0, 'ondisk_log_size': 8592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33263344, 'num_objects': 4061, 'num_object_clones': 0, 'num_object_copies': 16244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8589, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1760}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:21.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:21.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8655", 'reported_seq': 8747, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.650706+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:20.650706+0000', 'last_peered': '2024-09-11T05:02:20.650706+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:20.650706+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8655, 'log_dups_size': 0, 'ondisk_log_size': 8655, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33323600, 'num_objects': 4068, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8654, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1742}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:21.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:21.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8530", 'reported_seq': 8630, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:19.866089+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:19.866089+0000', 'last_peered': '2024-09-11T05:02:19.866089+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:19.866089+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8530, 'log_dups_size': 0, 'ondisk_log_size': 8530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33272992, 'num_objects': 4062, 'num_object_clones': 0, 'num_object_copies': 16248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2267, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8528, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:21.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:21.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8480", 'reported_seq': 8594, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.761536+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:20.761536+0000', 'last_peered': '2024-09-11T05:02:20.761536+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:20.761536+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8480, 'log_dups_size': 0, 'ondisk_log_size': 8480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32183456, 'num_objects': 3929, 'num_object_clones': 0, 'num_object_copies': 15716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8478, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1626}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:21.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:21.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8639", 'reported_seq': 8750, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:18.438452+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:18.438452+0000', 'last_peered': '2024-09-11T05:02:18.438452+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:18.438452+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8639, 'log_dups_size': 0, 'ondisk_log_size': 8639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33895584, 'num_objects': 4138, 'num_object_clones': 0, 'num_object_copies': 16552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8637, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1848}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:21.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:22.663 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:22.664 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32520/259032 objects degraded (12.554%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":469,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64758,"data_bytes":530450056,"bytes_used":2837716992,"bytes_avail":383709339648,"bytes_total":386547056640,"degraded_objects":32520,"degraded_total":259032,"degraded_ratio":0.12554433429074399,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":107135,"recovering_keys_per_sec":0,"num_objects_recovered":172,"num_bytes_recovered":1286144,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":231},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:22.701 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:22.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:23.318 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:23.345 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-11T05:02:21.773322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530450056,"num_objects":64758,"num_object_clones":0,"num_object_copies":259032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130435,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136651,"ondisk_log_size":136651,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771208,"kb_used_data":1035988,"kb_used_omap":14,"kb_used_meta":1735089,"kb_avail":374716152,"statfs":{"total":386547056640,"available":383709339648,"internally_reserved":0,"allocated":1060851712,"data_stored":1060131836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1776731702},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,13,8,14,1,0,0,0,1,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22806528,"num_objects":-2784,"num_object_clones":0,"num_object_copies":-11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004885"},"pg_stats":[{"pgid":"1.f","version":"33'8520","reported_seq":8609,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.010061+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:20.010061+0000","last_peered":"2024-09-11T05:02:20.010061+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:20.010061+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32718848,"num_objects":3994,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1703}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8580","reported_seq":8688,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.276719+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:18.276719+0000","last_peered":"2024-09-11T05:02:18.276719+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:18.276719+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8580,"log_dups_size":0,"ondisk_log_size":8580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059424,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1899}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8265","reported_seq":8363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.609113+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:20.609113+0000","last_peered":"2024-09-11T05:02:20.609113+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:20.609113+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31906408,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8264,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8406","reported_seq":8507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.311656+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:18.311656+0000","last_peered":"2024-09-11T05:02:18.311656+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:18.311656+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847008,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8472","reported_seq":8590,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.927334+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:19.927334+0000","last_peered":"2024-09-11T05:02:19.927334+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:19.927334+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8472,"log_dups_size":0,"ondisk_log_size":8472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33248416,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8470,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8599","reported_seq":8696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.952667+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:19.952667+0000","last_peered":"2024-09-11T05:02:19.952667+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:19.952667+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8599,"log_dups_size":0,"ondisk_log_size":8599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33103872,"num_objects":4041,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8589","reported_seq":8676,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.303605+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:18.303605+0000","last_peered":"2024-09-11T05:02:18.303605+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:18.303605+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33920160,"num_objects":4141,"num_object_clones":0,"num_object_copies":16564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8587,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8695","reported_seq":12508,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.312451+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:18.312451+0000","last_peered":"2024-09-11T05:02:18.312451+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:18.312451+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33830048,"num_objects":4130,"num_object_clones":0,"num_object_copies":16520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1874,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8737","reported_seq":8855,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.779078+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:19.779078+0000","last_peered":"2024-09-11T05:02:19.779078+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:19.779078+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8737,"log_dups_size":0,"ondisk_log_size":8737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33179968,"num_objects":4051,"num_object_clones":0,"num_object_copies":16204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8734,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8529","reported_seq":8636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.714620+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:20.714620+0000","last_peered":"2024-09-11T05:02:20.714620+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:20.714620+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756896,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8527,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8592","reported_seq":8698,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.984787+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:19.984787+0000","last_peered":"2024-09-11T05:02:19.984787+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:19.984787+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33263344,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8363","reported_seq":12959,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:19.310286+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:19.310286+0000","last_peered":"2024-09-11T05:02:19.310286+0000","last_clean":"2024-09-11T05:02:19.310286+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:19.310286+0000","last_undegraded":"2024-09-11T05:02:19.310286+0000","last_fullsized":"2024-09-11T05:02:19.310286+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8363,"log_dups_size":0,"ondisk_log_size":8363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32940032,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8655","reported_seq":8747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.650706+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:20.650706+0000","last_peered":"2024-09-11T05:02:20.650706+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:20.650706+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33323600,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1742}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8530","reported_seq":8630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:19.866089+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:19.866089+0000","last_peered":"2024-09-11T05:02:19.866089+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:19.866089+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33272992,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1797}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8480","reported_seq":8594,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.761536+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:20.761536+0000","last_peered":"2024-09-11T05:02:20.761536+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:20.761536+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183456,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1626}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8639","reported_seq":8750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:18.438452+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:18.438452+0000","last_peered":"2024-09-11T05:02:18.438452+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:18.438452+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33895584,"num_objects":4138,"num_object_clones":0,"num_object_copies":16552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8637,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530450056,"num_objects":64758,"num_object_clones":0,"num_object_copies":259032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130435,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":15278080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1057947648,"data_stored":1057947648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136651,"ondisk_log_size":136651,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719948,"kb_used_data":259240,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93651892,"statfs":{"total":96636764160,"available":95899537408,"internally_reserved":0,"allocated":265461760,"data_stored":265281791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,1,0,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731564,"kb_used_data":258876,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93640276,"statfs":{"total":96636764160,"available":95887642624,"internally_reserved":0,"allocated":265089024,"data_stored":264909055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589548,"kb_used_data":256668,"kb_used_omap":4,"kb_used_meta":332859,"kb_avail":93782292,"statfs":{"total":96636764160,"available":96033067008,"internally_reserved":0,"allocated":262828032,"data_stored":262648063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":340848338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730148,"kb_used_data":261204,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93641692,"statfs":{"total":96636764160,"available":95889092608,"internally_reserved":0,"allocated":267472896,"data_stored":267292927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,1,4,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266362880,"data_stored":266362880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262496256,"data_stored":262496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264196096,"data_stored":264196096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":264892416,"data_stored":264892416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:26.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:26.989 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:27.007 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-11T05:02:25.777941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523978376,"num_objects":63968,"num_object_clones":0,"num_object_copies":255872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131225,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137441,"ondisk_log_size":137441,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757872,"kb_used_data":1026832,"kb_used_omap":14,"kb_used_meta":1730929,"kb_avail":374729488,"statfs":{"total":386547056640,"available":383722995712,"internally_reserved":0,"allocated":1051475968,"data_stored":1050756092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1772471862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,9,1,7,15,20,4,0,0,0,0,2,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21078016,"num_objects":-2573,"num_object_clones":0,"num_object_copies":-10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007701"},"pg_stats":[{"pgid":"1.f","version":"33'8577","reported_seq":8667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.905492+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:24.905492+0000","last_peered":"2024-09-11T05:02:24.905492+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:24.905492+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8577,"log_dups_size":0,"ondisk_log_size":8577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8633","reported_seq":8741,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.161244+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:23.161244+0000","last_peered":"2024-09-11T05:02:23.161244+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:23.161244+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8633,"log_dups_size":0,"ondisk_log_size":8633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33625248,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8265","reported_seq":8363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.609113+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:20.609113+0000","last_peered":"2024-09-11T05:02:20.609113+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:20.609113+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31906408,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8264,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8480","reported_seq":8581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.426433+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:23.426433+0000","last_peered":"2024-09-11T05:02:23.426433+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:23.426433+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32240800,"num_objects":3936,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1674}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8535","reported_seq":8651,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:25.056430+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:25.056430+0000","last_peered":"2024-09-11T05:02:25.056430+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:25.056430+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8535,"log_dups_size":0,"ondisk_log_size":8535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32732320,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8533,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8657","reported_seq":8756,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.979558+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:24.979558+0000","last_peered":"2024-09-11T05:02:24.979558+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:24.979558+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32628736,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8657,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8671","reported_seq":8757,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.427063+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:23.427063+0000","last_peered":"2024-09-11T05:02:23.427063+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:23.427063+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33248416,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8764","reported_seq":12721,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.412922+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:23.412922+0000","last_peered":"2024-09-11T05:02:23.412922+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:23.412922+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264800,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8802","reported_seq":8920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.810711+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:24.810711+0000","last_peered":"2024-09-11T05:02:24.810711+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:24.810711+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32647488,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8799,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8529","reported_seq":8636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.714620+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:20.714620+0000","last_peered":"2024-09-11T05:02:20.714620+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:20.714620+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756896,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8527,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8665","reported_seq":8775,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.980318+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:24.980318+0000","last_peered":"2024-09-11T05:02:24.980318+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:24.980318+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8665,"log_dups_size":0,"ondisk_log_size":8665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32665328,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8431","reported_seq":13027,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:24.178099+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:24.178099+0000","last_peered":"2024-09-11T05:02:24.178099+0000","last_clean":"2024-09-11T05:02:24.178099+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:24.178099+0000","last_undegraded":"2024-09-11T05:02:24.178099+0000","last_fullsized":"2024-09-11T05:02:24.178099+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8431,"log_dups_size":0,"ondisk_log_size":8431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382976,"num_objects":3953,"num_object_clones":0,"num_object_copies":15812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8655","reported_seq":8747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.650706+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:20.650706+0000","last_peered":"2024-09-11T05:02:20.650706+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:20.650706+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33323600,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1742}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8600","reported_seq":8702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.861668+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:24.861668+0000","last_peered":"2024-09-11T05:02:24.861668+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:24.861668+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699552,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8598,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8480","reported_seq":8594,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.761536+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:20.761536+0000","last_peered":"2024-09-11T05:02:20.761536+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:20.761536+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183456,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1626}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8697","reported_seq":8808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.413300+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:23.413300+0000","last_peered":"2024-09-11T05:02:23.413300+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:23.413300+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33420448,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":523978376,"num_objects":63968,"num_object_clones":0,"num_object_copies":255872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131225,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048256512,"data_stored":1048256512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137441,"ondisk_log_size":137441,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719788,"kb_used_data":254940,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93652052,"statfs":{"total":96636764160,"available":95899701248,"internally_reserved":0,"allocated":261058560,"data_stored":260878591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,4,4,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731564,"kb_used_data":258876,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93640276,"statfs":{"total":96636764160,"available":95887642624,"internally_reserved":0,"allocated":265089024,"data_stored":264909055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576596,"kb_used_data":256196,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93795244,"statfs":{"total":96636764160,"available":96046329856,"internally_reserved":0,"allocated":262344704,"data_stored":262164735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328068818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729924,"kb_used_data":256820,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93641916,"statfs":{"total":96636764160,"available":95889321984,"internally_reserved":0,"allocated":262983680,"data_stored":262803711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261910528,"data_stored":261910528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261738496,"data_stored":261738496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264196096,"data_stored":264196096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260411392,"data_stored":260411392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:27.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:27.685 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:27.706 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-11T05:02:25.777941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523978376,"num_objects":63968,"num_object_clones":0,"num_object_copies":255872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131225,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137441,"ondisk_log_size":137441,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757872,"kb_used_data":1026832,"kb_used_omap":14,"kb_used_meta":1730929,"kb_avail":374729488,"statfs":{"total":386547056640,"available":383722995712,"internally_reserved":0,"allocated":1051475968,"data_stored":1050756092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1772471862},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,9,1,7,15,20,4,0,0,0,0,2,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21078016,"num_objects":-2573,"num_object_clones":0,"num_object_copies":-10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007701"},"pg_stats":[{"pgid":"1.f","version":"33'8577","reported_seq":8667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.905492+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:24.905492+0000","last_peered":"2024-09-11T05:02:24.905492+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:24.905492+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8577,"log_dups_size":0,"ondisk_log_size":8577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8633","reported_seq":8741,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.161244+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:23.161244+0000","last_peered":"2024-09-11T05:02:23.161244+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:23.161244+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8633,"log_dups_size":0,"ondisk_log_size":8633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33625248,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8265","reported_seq":8363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.609113+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:20.609113+0000","last_peered":"2024-09-11T05:02:20.609113+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:20.609113+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8265,"log_dups_size":0,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31906408,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8264,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8480","reported_seq":8581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.426433+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:23.426433+0000","last_peered":"2024-09-11T05:02:23.426433+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:23.426433+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32240800,"num_objects":3936,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1674}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8535","reported_seq":8651,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:25.056430+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:25.056430+0000","last_peered":"2024-09-11T05:02:25.056430+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:25.056430+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8535,"log_dups_size":0,"ondisk_log_size":8535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32732320,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8533,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8657","reported_seq":8756,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.979558+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:24.979558+0000","last_peered":"2024-09-11T05:02:24.979558+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:24.979558+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32628736,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8657,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8671","reported_seq":8757,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.427063+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:23.427063+0000","last_peered":"2024-09-11T05:02:23.427063+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:23.427063+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33248416,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8764","reported_seq":12721,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.412922+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:23.412922+0000","last_peered":"2024-09-11T05:02:23.412922+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:23.412922+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264800,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8802","reported_seq":8920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.810711+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:24.810711+0000","last_peered":"2024-09-11T05:02:24.810711+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:24.810711+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32647488,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8799,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8529","reported_seq":8636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.714620+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:20.714620+0000","last_peered":"2024-09-11T05:02:20.714620+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:20.714620+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756896,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8527,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1693}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8665","reported_seq":8775,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.980318+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:24.980318+0000","last_peered":"2024-09-11T05:02:24.980318+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:24.980318+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8665,"log_dups_size":0,"ondisk_log_size":8665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32665328,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8431","reported_seq":13027,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:24.178099+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:24.178099+0000","last_peered":"2024-09-11T05:02:24.178099+0000","last_clean":"2024-09-11T05:02:24.178099+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:24.178099+0000","last_undegraded":"2024-09-11T05:02:24.178099+0000","last_fullsized":"2024-09-11T05:02:24.178099+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8431,"log_dups_size":0,"ondisk_log_size":8431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382976,"num_objects":3953,"num_object_clones":0,"num_object_copies":15812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8655","reported_seq":8747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.650706+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:20.650706+0000","last_peered":"2024-09-11T05:02:20.650706+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:20.650706+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33323600,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1742}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8600","reported_seq":8702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.861668+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:24.861668+0000","last_peered":"2024-09-11T05:02:24.861668+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:24.861668+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699552,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8598,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8480","reported_seq":8594,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:20.761536+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:20.761536+0000","last_peered":"2024-09-11T05:02:20.761536+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:20.761536+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183456,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1626}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8697","reported_seq":8808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.413300+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:23.413300+0000","last_peered":"2024-09-11T05:02:23.413300+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:23.413300+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33420448,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":523978376,"num_objects":63968,"num_object_clones":0,"num_object_copies":255872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131225,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048256512,"data_stored":1048256512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137441,"ondisk_log_size":137441,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719788,"kb_used_data":254940,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93652052,"statfs":{"total":96636764160,"available":95899701248,"internally_reserved":0,"allocated":261058560,"data_stored":260878591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,4,4,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731564,"kb_used_data":258876,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93640276,"statfs":{"total":96636764160,"available":95887642624,"internally_reserved":0,"allocated":265089024,"data_stored":264909055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576596,"kb_used_data":256196,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93795244,"statfs":{"total":96636764160,"available":96046329856,"internally_reserved":0,"allocated":262344704,"data_stored":262164735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328068818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729924,"kb_used_data":256820,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93641916,"statfs":{"total":96636764160,"available":95889321984,"internally_reserved":0,"allocated":262983680,"data_stored":262803711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261910528,"data_stored":261910528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261738496,"data_stored":261738496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264196096,"data_stored":264196096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260411392,"data_stored":260411392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:27.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:27.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8577", 'reported_seq': 8667, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:24.905492+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:24.905492+0000', 'last_peered': '2024-09-11T05:02:24.905492+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:24.905492+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8577, 'log_dups_size': 0, 'ondisk_log_size': 8577, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32251904, 'num_objects': 3937, 'num_object_clones': 0, 'num_object_copies': 15748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8577, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1646}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:27.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:27.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8633", 'reported_seq': 8741, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:23.161244+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:23.161244+0000', 'last_peered': '2024-09-11T05:02:23.161244+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:23.161244+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8633, 'log_dups_size': 0, 'ondisk_log_size': 8633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33625248, 'num_objects': 4105, 'num_object_clones': 0, 'num_object_copies': 16420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2261, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8631, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1846}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8265", 'reported_seq': 8363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.609113+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:20.609113+0000', 'last_peered': '2024-09-11T05:02:20.609113+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:20.609113+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8265, 'log_dups_size': 0, 'ondisk_log_size': 8265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31906408, 'num_objects': 3896, 'num_object_clones': 0, 'num_object_copies': 15584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3896, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8264, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1685}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8480", 'reported_seq': 8581, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:23.426433+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:23.426433+0000', 'last_peered': '2024-09-11T05:02:23.426433+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:23.426433+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8480, 'log_dups_size': 0, 'ondisk_log_size': 8480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32240800, 'num_objects': 3936, 'num_object_clones': 0, 'num_object_copies': 15744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8478, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1674}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8535", 'reported_seq': 8651, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:25.056430+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:25.056430+0000', 'last_peered': '2024-09-11T05:02:25.056430+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:25.056430+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8535, 'log_dups_size': 0, 'ondisk_log_size': 8535, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32732320, 'num_objects': 3996, 'num_object_clones': 0, 'num_object_copies': 15984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8533, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1775}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8657", 'reported_seq': 8756, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:24.979558+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:24.979558+0000', 'last_peered': '2024-09-11T05:02:24.979558+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:24.979558+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8657, 'log_dups_size': 0, 'ondisk_log_size': 8657, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32628736, 'num_objects': 3983, 'num_object_clones': 0, 'num_object_copies': 15932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8657, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1688}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:27.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8671", 'reported_seq': 8757, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:23.427063+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:23.427063+0000', 'last_peered': '2024-09-11T05:02:23.427063+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:23.427063+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8671, 'log_dups_size': 0, 'ondisk_log_size': 8671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33248416, 'num_objects': 4059, 'num_object_clones': 0, 'num_object_copies': 16236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8669, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1751}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8764", 'reported_seq': 12721, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:23.412922+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:23.412922+0000', 'last_peered': '2024-09-11T05:02:23.412922+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:23.412922+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8764, 'log_dups_size': 0, 'ondisk_log_size': 8764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33264800, 'num_objects': 4061, 'num_object_clones': 0, 'num_object_copies': 16244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 419, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8762, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1946, 'num_bytes_recovered': 15564800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3642}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8802", 'reported_seq': 8920, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:24.810711+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:24.810711+0000', 'last_peered': '2024-09-11T05:02:24.810711+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:24.810711+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8802, 'log_dups_size': 0, 'ondisk_log_size': 8802, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32647488, 'num_objects': 3986, 'num_object_clones': 0, 'num_object_copies': 15944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8799, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1620}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8529", 'reported_seq': 8636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.714620+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:20.714620+0000', 'last_peered': '2024-09-11T05:02:20.714620+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:20.714620+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8529, 'log_dups_size': 0, 'ondisk_log_size': 8529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32756896, 'num_objects': 3999, 'num_object_clones': 0, 'num_object_copies': 15996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8527, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1693}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8665", 'reported_seq': 8775, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:24.980318+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:24.980318+0000', 'last_peered': '2024-09-11T05:02:24.980318+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:24.980318+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8665, 'log_dups_size': 0, 'ondisk_log_size': 8665, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32665328, 'num_objects': 3988, 'num_object_clones': 0, 'num_object_copies': 15952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2302, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8662, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1689}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:27.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8655", 'reported_seq': 8747, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.650706+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:20.650706+0000', 'last_peered': '2024-09-11T05:02:20.650706+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:20.650706+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8655, 'log_dups_size': 0, 'ondisk_log_size': 8655, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33323600, 'num_objects': 4068, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8654, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1742}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:27.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:27.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8600", 'reported_seq': 8702, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:24.861668+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:24.861668+0000', 'last_peered': '2024-09-11T05:02:24.861668+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:24.861668+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8600, 'log_dups_size': 0, 'ondisk_log_size': 8600, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32699552, 'num_objects': 3992, 'num_object_clones': 0, 'num_object_copies': 15968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8598, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1728}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:27.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:27.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8480", 'reported_seq': 8594, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:20.761536+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:20.761536+0000', 'last_peered': '2024-09-11T05:02:20.761536+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:20.761536+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8480, 'log_dups_size': 0, 'ondisk_log_size': 8480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32183456, 'num_objects': 3929, 'num_object_clones': 0, 'num_object_copies': 15716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8478, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1626}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:27.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:27.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8697", 'reported_seq': 8808, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:23.413300+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:23.413300+0000', 'last_peered': '2024-09-11T05:02:23.413300+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:23.413300+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8697, 'log_dups_size': 0, 'ondisk_log_size': 8697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33420448, 'num_objects': 4080, 'num_object_clones': 0, 'num_object_copies': 16320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8695, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:27.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:28.499 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:28.499 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32443/254784 objects degraded (12.734%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":475,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63696,"data_bytes":521750152,"bytes_used":2806747136,"bytes_avail":383740309504,"bytes_total":386547056640,"degraded_objects":32443,"degraded_total":254784,"degraded_ratio":0.12733531147952776,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":77091,"recovering_keys_per_sec":0,"num_objects_recovered":163,"num_bytes_recovered":925696,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":179},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:28.523 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:28.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:29.200 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:29.226 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-11T05:02:27.778818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":521750152,"num_objects":63696,"num_object_clones":0,"num_object_copies":254784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131497,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137713,"ondisk_log_size":137713,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740964,"kb_used_data":1022436,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374746396,"statfs":{"total":386547056640,"available":383740309504,"internally_reserved":0,"allocated":1046974464,"data_stored":1046254588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1759692342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,6,2,11,14,3,0,0,0,0,2,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17678336,"num_objects":-2158,"num_object_clones":0,"num_object_copies":-8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2158,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007806"},"pg_stats":[{"pgid":"1.f","version":"33'8577","reported_seq":8667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.905492+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:24.905492+0000","last_peered":"2024-09-11T05:02:24.905492+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:24.905492+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8577,"log_dups_size":0,"ondisk_log_size":8577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32251904,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8633","reported_seq":8741,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.161244+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:23.161244+0000","last_peered":"2024-09-11T05:02:23.161244+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:23.161244+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8633,"log_dups_size":0,"ondisk_log_size":8633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33625248,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8324","reported_seq":8424,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:25.773615+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:25.773615+0000","last_peered":"2024-09-11T05:02:25.773615+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:25.773615+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31423080,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8323,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1626}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8480","reported_seq":8581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.426433+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:23.426433+0000","last_peered":"2024-09-11T05:02:23.426433+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:23.426433+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32240800,"num_objects":3936,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1674}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8535","reported_seq":8651,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:25.056430+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:25.056430+0000","last_peered":"2024-09-11T05:02:25.056430+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:25.056430+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8535,"log_dups_size":0,"ondisk_log_size":8535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32732320,"num_objects":3996,"num_object_clones":0,"num_object_copies":15984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8533,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8657","reported_seq":8756,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.979558+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:24.979558+0000","last_peered":"2024-09-11T05:02:24.979558+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:24.979558+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32628736,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8657,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8671","reported_seq":8757,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.427063+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:23.427063+0000","last_peered":"2024-09-11T05:02:23.427063+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:23.427063+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33248416,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8764","reported_seq":12721,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.412922+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:23.412922+0000","last_peered":"2024-09-11T05:02:23.412922+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:23.412922+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264800,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8802","reported_seq":8920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.810711+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:24.810711+0000","last_peered":"2024-09-11T05:02:24.810711+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:24.810711+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32647488,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8799,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8609","reported_seq":8716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:25.627608+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:25.627608+0000","last_peered":"2024-09-11T05:02:25.627608+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:25.627608+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8609,"log_dups_size":0,"ondisk_log_size":8609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32101536,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8665","reported_seq":8775,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.980318+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:24.980318+0000","last_peered":"2024-09-11T05:02:24.980318+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:24.980318+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8665,"log_dups_size":0,"ondisk_log_size":8665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32665328,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8431","reported_seq":13027,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:24.178099+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:24.178099+0000","last_peered":"2024-09-11T05:02:24.178099+0000","last_clean":"2024-09-11T05:02:24.178099+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:24.178099+0000","last_undegraded":"2024-09-11T05:02:24.178099+0000","last_fullsized":"2024-09-11T05:02:24.178099+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8431,"log_dups_size":0,"ondisk_log_size":8431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382976,"num_objects":3953,"num_object_clones":0,"num_object_copies":15812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8724","reported_seq":8816,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:25.764773+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:25.764773+0000","last_peered":"2024-09-11T05:02:25.764773+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:25.764773+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8724,"log_dups_size":0,"ondisk_log_size":8724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32758352,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8723,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1673}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8600","reported_seq":8702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:24.861668+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:24.861668+0000","last_peered":"2024-09-11T05:02:24.861668+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:24.861668+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699552,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8598,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8544","reported_seq":8658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:25.516796+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:25.516796+0000","last_peered":"2024-09-11T05:02:25.516796+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:25.516796+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31659168,"num_objects":3865,"num_object_clones":0,"num_object_copies":15460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8542,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8697","reported_seq":8808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:23.413300+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:23.413300+0000","last_peered":"2024-09-11T05:02:23.413300+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:23.413300+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33420448,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":521750152,"num_objects":63696,"num_object_clones":0,"num_object_copies":254784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131497,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1043812352,"data_stored":1043812352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137713,"ondisk_log_size":137713,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719788,"kb_used_data":254940,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93652052,"statfs":{"total":96636764160,"available":95899701248,"internally_reserved":0,"allocated":261058560,"data_stored":260878591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,4,4,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714656,"kb_used_data":254480,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93657184,"statfs":{"total":96636764160,"available":95904956416,"internally_reserved":0,"allocated":260587520,"data_stored":260407551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576596,"kb_used_data":256196,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93795244,"statfs":{"total":96636764160,"available":96046329856,"internally_reserved":0,"allocated":262344704,"data_stored":262164735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":328068818},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729924,"kb_used_data":256820,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93641916,"statfs":{"total":96636764160,"available":95889321984,"internally_reserved":0,"allocated":262983680,"data_stored":262803711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261910528,"data_stored":261910528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261738496,"data_stored":261738496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259751936,"data_stored":259751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260411392,"data_stored":260411392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:32.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:32.834 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:32.834 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-11T05:02:31.780911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512755336,"num_objects":62598,"num_object_clones":0,"num_object_copies":250392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132595,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138811,"ondisk_log_size":138811,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2723904,"kb_used_data":1005332,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374763456,"statfs":{"total":386547056640,"available":383757778944,"internally_reserved":0,"allocated":1029459968,"data_stored":1028740092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1759692342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,9,10,13,17,9,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23511040,"num_objects":-2870,"num_object_clones":0,"num_object_copies":-11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":286720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008306"},"pg_stats":[{"pgid":"1.f","version":"33'8649","reported_seq":8738,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.121725+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:30.121725+0000","last_peered":"2024-09-11T05:02:30.121725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:30.121725+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31662080,"num_objects":3865,"num_object_clones":0,"num_object_copies":15460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8720","reported_seq":8828,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.491914+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:28.491914+0000","last_peered":"2024-09-11T05:02:28.491914+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:28.491914+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8720,"log_dups_size":0,"ondisk_log_size":8720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32912544,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8718,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8387","reported_seq":8487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.831897+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:30.831897+0000","last_peered":"2024-09-11T05:02:30.831897+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:30.831897+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30906984,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8386,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1563}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8560","reported_seq":8663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.392269+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:28.392269+0000","last_peered":"2024-09-11T05:02:28.392269+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:28.392269+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31585440,"num_objects":3856,"num_object_clones":0,"num_object_copies":15424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8558,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8601","reported_seq":8719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.039806+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:30.039806+0000","last_peered":"2024-09-11T05:02:30.039806+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:30.039806+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32191648,"num_objects":3930,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8740","reported_seq":8841,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:29.979105+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:29.979105+0000","last_peered":"2024-09-11T05:02:29.979105+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:29.979105+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31948800,"num_objects":3900,"num_object_clones":0,"num_object_copies":15600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8740","reported_seq":8829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.415250+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:28.415250+0000","last_peered":"2024-09-11T05:02:28.415250+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:28.415250+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32683168,"num_objects":3990,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1683}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8836","reported_seq":12905,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.513463+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:28.513463+0000","last_peered":"2024-09-11T05:02:28.513463+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:28.513463+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32674976,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3626}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8854","reported_seq":8972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:29.978986+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:29.978986+0000","last_peered":"2024-09-11T05:02:29.978986+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:29.978986+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32221504,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8673","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.781836+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:30.781836+0000","last_peered":"2024-09-11T05:02:30.781836+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:30.781836+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31577248,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8671,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8734","reported_seq":8846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.030787+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:30.030787+0000","last_peered":"2024-09-11T05:02:30.030787+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:30.030787+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32100080,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8497","reported_seq":13093,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:29.404349+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:29.404349+0000","last_peered":"2024-09-11T05:02:29.404349+0000","last_clean":"2024-09-11T05:02:29.404349+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:29.404349+0000","last_undegraded":"2024-09-11T05:02:29.404349+0000","last_fullsized":"2024-09-11T05:02:29.404349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8497,"log_dups_size":0,"ondisk_log_size":8497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31842304,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2307,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8782","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.508198+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:30.508198+0000","last_peered":"2024-09-11T05:02:30.508198+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:30.508198+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8782,"log_dups_size":0,"ondisk_log_size":8782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32283216,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1615}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8664","reported_seq":8765,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.064140+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:30.064140+0000","last_peered":"2024-09-11T05:02:30.064140+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:30.064140+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32175264,"num_objects":3928,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1664}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8607","reported_seq":8718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.712515+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:30.712515+0000","last_peered":"2024-09-11T05:02:30.712515+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:30.712515+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8607,"log_dups_size":0,"ondisk_log_size":8607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8605,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8767","reported_seq":8882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.512471+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:28.512471+0000","last_peered":"2024-09-11T05:02:28.512471+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:28.512471+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8767,"log_dups_size":0,"ondisk_log_size":8767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847008,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8765,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1722}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":512755336,"num_objects":62598,"num_object_clones":0,"num_object_copies":250392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132595,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1025929216,"data_stored":1025929216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138811,"ondisk_log_size":138811,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719620,"kb_used_data":250608,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93652220,"statfs":{"total":96636764160,"available":95899873280,"internally_reserved":0,"allocated":256622592,"data_stored":256442623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,1,3,2,3,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714460,"kb_used_data":250124,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93657380,"statfs":{"total":96636764160,"available":95905157120,"internally_reserved":0,"allocated":256126976,"data_stored":255947007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576744,"kb_used_data":252164,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93795096,"statfs":{"total":96636764160,"available":96046178304,"internally_reserved":0,"allocated":258215936,"data_stored":258035967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332328658},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713080,"kb_used_data":252436,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93658760,"statfs":{"total":96636764160,"available":95906570240,"internally_reserved":0,"allocated":258494464,"data_stored":258314495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,3,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257449984,"data_stored":257449984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257228800,"data_stored":257228800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":255238144,"data_stored":255238144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256012288,"data_stored":256012288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:32.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:33.568 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:33.568 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:33.591 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-11T05:02:31.780911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512755336,"num_objects":62598,"num_object_clones":0,"num_object_copies":250392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132595,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138811,"ondisk_log_size":138811,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2723904,"kb_used_data":1005332,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374763456,"statfs":{"total":386547056640,"available":383757778944,"internally_reserved":0,"allocated":1029459968,"data_stored":1028740092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1759692342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,9,10,13,17,9,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23511040,"num_objects":-2870,"num_object_clones":0,"num_object_copies":-11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":286720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008306"},"pg_stats":[{"pgid":"1.f","version":"33'8649","reported_seq":8738,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.121725+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:30.121725+0000","last_peered":"2024-09-11T05:02:30.121725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:30.121725+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31662080,"num_objects":3865,"num_object_clones":0,"num_object_copies":15460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8720","reported_seq":8828,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.491914+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:28.491914+0000","last_peered":"2024-09-11T05:02:28.491914+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:28.491914+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8720,"log_dups_size":0,"ondisk_log_size":8720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32912544,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8718,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8387","reported_seq":8487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.831897+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:30.831897+0000","last_peered":"2024-09-11T05:02:30.831897+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:30.831897+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30906984,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8386,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1563}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8560","reported_seq":8663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.392269+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:28.392269+0000","last_peered":"2024-09-11T05:02:28.392269+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:28.392269+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31585440,"num_objects":3856,"num_object_clones":0,"num_object_copies":15424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8558,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8601","reported_seq":8719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.039806+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:30.039806+0000","last_peered":"2024-09-11T05:02:30.039806+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:30.039806+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32191648,"num_objects":3930,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8740","reported_seq":8841,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:29.979105+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:29.979105+0000","last_peered":"2024-09-11T05:02:29.979105+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:29.979105+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31948800,"num_objects":3900,"num_object_clones":0,"num_object_copies":15600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8740","reported_seq":8829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.415250+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:28.415250+0000","last_peered":"2024-09-11T05:02:28.415250+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:28.415250+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32683168,"num_objects":3990,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1683}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8836","reported_seq":12905,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.513463+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:28.513463+0000","last_peered":"2024-09-11T05:02:28.513463+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:28.513463+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32674976,"num_objects":3989,"num_object_clones":0,"num_object_copies":15956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3626}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8854","reported_seq":8972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:29.978986+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:29.978986+0000","last_peered":"2024-09-11T05:02:29.978986+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:29.978986+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32221504,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8673","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.781836+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:30.781836+0000","last_peered":"2024-09-11T05:02:30.781836+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:30.781836+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31577248,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8671,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8734","reported_seq":8846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.030787+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:30.030787+0000","last_peered":"2024-09-11T05:02:30.030787+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:30.030787+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32100080,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8497","reported_seq":13093,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:29.404349+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:29.404349+0000","last_peered":"2024-09-11T05:02:29.404349+0000","last_clean":"2024-09-11T05:02:29.404349+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:29.404349+0000","last_undegraded":"2024-09-11T05:02:29.404349+0000","last_fullsized":"2024-09-11T05:02:29.404349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8497,"log_dups_size":0,"ondisk_log_size":8497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31842304,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2307,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8782","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.508198+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:30.508198+0000","last_peered":"2024-09-11T05:02:30.508198+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:30.508198+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8782,"log_dups_size":0,"ondisk_log_size":8782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32283216,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1615}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8664","reported_seq":8765,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.064140+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:30.064140+0000","last_peered":"2024-09-11T05:02:30.064140+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:30.064140+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32175264,"num_objects":3928,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1664}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8607","reported_seq":8718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.712515+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:30.712515+0000","last_peered":"2024-09-11T05:02:30.712515+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:30.712515+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8607,"log_dups_size":0,"ondisk_log_size":8607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8605,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8767","reported_seq":8882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:28.512471+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:28.512471+0000","last_peered":"2024-09-11T05:02:28.512471+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:28.512471+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8767,"log_dups_size":0,"ondisk_log_size":8767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847008,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8765,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1722}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":512755336,"num_objects":62598,"num_object_clones":0,"num_object_copies":250392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132595,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1025929216,"data_stored":1025929216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138811,"ondisk_log_size":138811,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719620,"kb_used_data":250608,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93652220,"statfs":{"total":96636764160,"available":95899873280,"internally_reserved":0,"allocated":256622592,"data_stored":256442623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,1,3,2,3,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714460,"kb_used_data":250124,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93657380,"statfs":{"total":96636764160,"available":95905157120,"internally_reserved":0,"allocated":256126976,"data_stored":255947007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576744,"kb_used_data":252164,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93795096,"statfs":{"total":96636764160,"available":96046178304,"internally_reserved":0,"allocated":258215936,"data_stored":258035967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332328658},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713080,"kb_used_data":252436,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93658760,"statfs":{"total":96636764160,"available":95906570240,"internally_reserved":0,"allocated":258494464,"data_stored":258314495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,3,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257449984,"data_stored":257449984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257228800,"data_stored":257228800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":255238144,"data_stored":255238144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256012288,"data_stored":256012288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:33.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:33.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8649", 'reported_seq': 8738, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:30.121725+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:30.121725+0000', 'last_peered': '2024-09-11T05:02:30.121725+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:30.121725+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8649, 'log_dups_size': 0, 'ondisk_log_size': 8649, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31662080, 'num_objects': 3865, 'num_object_clones': 0, 'num_object_copies': 15460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8649, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1574}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:33.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8720", 'reported_seq': 8828, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:28.491914+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:28.491914+0000', 'last_peered': '2024-09-11T05:02:28.491914+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:28.491914+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8720, 'log_dups_size': 0, 'ondisk_log_size': 8720, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32912544, 'num_objects': 4018, 'num_object_clones': 0, 'num_object_copies': 16072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8718, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8387", 'reported_seq': 8487, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:30.831897+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:30.831897+0000', 'last_peered': '2024-09-11T05:02:30.831897+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:30.831897+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8387, 'log_dups_size': 0, 'ondisk_log_size': 8387, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30906984, 'num_objects': 3774, 'num_object_clones': 0, 'num_object_copies': 15096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3774, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8386, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1563}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8560", 'reported_seq': 8663, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:28.392269+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:28.392269+0000', 'last_peered': '2024-09-11T05:02:28.392269+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:28.392269+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8560, 'log_dups_size': 0, 'ondisk_log_size': 8560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31585440, 'num_objects': 3856, 'num_object_clones': 0, 'num_object_copies': 15424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8558, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1595}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8601", 'reported_seq': 8719, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:30.039806+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:30.039806+0000', 'last_peered': '2024-09-11T05:02:30.039806+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:30.039806+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8601, 'log_dups_size': 0, 'ondisk_log_size': 8601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32191648, 'num_objects': 3930, 'num_object_clones': 0, 'num_object_copies': 15720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8599, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1709}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8740", 'reported_seq': 8841, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:29.979105+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:29.979105+0000', 'last_peered': '2024-09-11T05:02:29.979105+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:29.979105+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8740, 'log_dups_size': 0, 'ondisk_log_size': 8740, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31948800, 'num_objects': 3900, 'num_object_clones': 0, 'num_object_copies': 15600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8740, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1606}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8740", 'reported_seq': 8829, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:28.415250+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:28.415250+0000', 'last_peered': '2024-09-11T05:02:28.415250+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:28.415250+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8740, 'log_dups_size': 0, 'ondisk_log_size': 8740, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32683168, 'num_objects': 3990, 'num_object_clones': 0, 'num_object_copies': 15960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8738, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1683}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8836", 'reported_seq': 12905, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:28.513463+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:28.513463+0000', 'last_peered': '2024-09-11T05:02:28.513463+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:28.513463+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8836, 'log_dups_size': 0, 'ondisk_log_size': 8836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32674976, 'num_objects': 3989, 'num_object_clones': 0, 'num_object_copies': 15956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8834, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2002, 'num_bytes_recovered': 15564800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3626}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8854", 'reported_seq': 8972, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:29.978986+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:29.978986+0000', 'last_peered': '2024-09-11T05:02:29.978986+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:29.978986+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8854, 'log_dups_size': 0, 'ondisk_log_size': 8854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32221504, 'num_objects': 3934, 'num_object_clones': 0, 'num_object_copies': 15736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8851, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1568}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8673", 'reported_seq': 8780, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:30.781836+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:30.781836+0000', 'last_peered': '2024-09-11T05:02:30.781836+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:30.781836+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8673, 'log_dups_size': 0, 'ondisk_log_size': 8673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31577248, 'num_objects': 3855, 'num_object_clones': 0, 'num_object_copies': 15420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8671, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1549}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8734", 'reported_seq': 8846, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:30.030787+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:30.030787+0000', 'last_peered': '2024-09-11T05:02:30.030787+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:30.030787+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8734, 'log_dups_size': 0, 'ondisk_log_size': 8734, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32100080, 'num_objects': 3919, 'num_object_clones': 0, 'num_object_copies': 15676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8731, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1621}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8782", 'reported_seq': 8874, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:30.508198+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:30.508198+0000', 'last_peered': '2024-09-11T05:02:30.508198+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:30.508198+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8782, 'log_dups_size': 0, 'ondisk_log_size': 8782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32283216, 'num_objects': 3941, 'num_object_clones': 0, 'num_object_copies': 15764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8781, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:33.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:33.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8664", 'reported_seq': 8765, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:30.064140+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:30.064140+0000', 'last_peered': '2024-09-11T05:02:30.064140+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:30.064140+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8664, 'log_dups_size': 0, 'ondisk_log_size': 8664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32175264, 'num_objects': 3928, 'num_object_clones': 0, 'num_object_copies': 15712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8662, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1664}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:33.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:33.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8607", 'reported_seq': 8718, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:30.712515+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:30.712515+0000', 'last_peered': '2024-09-11T05:02:30.712515+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:30.712515+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8607, 'log_dups_size': 0, 'ondisk_log_size': 8607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31143072, 'num_objects': 3802, 'num_object_clones': 0, 'num_object_copies': 15208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8605, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1499}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:33.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:33.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8767", 'reported_seq': 8882, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:28.512471+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:28.512471+0000', 'last_peered': '2024-09-11T05:02:28.512471+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:28.512471+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8767, 'log_dups_size': 0, 'ondisk_log_size': 8767, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32847008, 'num_objects': 4010, 'num_object_clones': 0, 'num_object_copies': 16040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8765, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1722}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:33.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:34.383 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:34.383 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32322/248912 objects degraded (12.985%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":481,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62228,"data_bytes":509724296,"bytes_used":2792497152,"bytes_avail":383754559488,"bytes_total":386547056640,"degraded_objects":32322,"degraded_total":248912,"degraded_ratio":0.12985312078164171,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":23875,"recovering_keys_per_sec":0,"num_objects_recovered":198,"num_bytes_recovered":286720,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:34.411 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:34.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:35.083 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:35.084 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:35.108 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-11T05:02:33.782482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":509724296,"num_objects":62228,"num_object_clones":0,"num_object_copies":248912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132965,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139181,"ondisk_log_size":139181,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727048,"kb_used_data":1000176,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374760312,"statfs":{"total":386547056640,"available":383754559488,"internally_reserved":0,"allocated":1024180224,"data_stored":1023460348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1768212022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,9,10,13,17,9,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20725760,"num_objects":-2530,"num_object_clones":0,"num_object_copies":-10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":286720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009160"},"pg_stats":[{"pgid":"1.f","version":"33'8649","reported_seq":8738,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.121725+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:30.121725+0000","last_peered":"2024-09-11T05:02:30.121725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:30.121725+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31662080,"num_objects":3865,"num_object_clones":0,"num_object_copies":15460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8786","reported_seq":8896,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.583401+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:33.583401+0000","last_peered":"2024-09-11T05:02:33.583401+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:33.583401+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32371872,"num_objects":3952,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8387","reported_seq":8487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.831897+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:30.831897+0000","last_peered":"2024-09-11T05:02:30.831897+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:30.831897+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30906984,"num_objects":3774,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8386,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1563}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8634","reported_seq":8737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.396238+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:33.396238+0000","last_peered":"2024-09-11T05:02:33.396238+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:33.396238+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8634,"log_dups_size":0,"ondisk_log_size":8634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30979232,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8632,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8601","reported_seq":8719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.039806+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:30.039806+0000","last_peered":"2024-09-11T05:02:30.039806+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:30.039806+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32191648,"num_objects":3930,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8740","reported_seq":8841,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:29.979105+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:29.979105+0000","last_peered":"2024-09-11T05:02:29.979105+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:29.979105+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31948800,"num_objects":3900,"num_object_clones":0,"num_object_copies":15600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8818","reported_seq":8907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.562898+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:33.562898+0000","last_peered":"2024-09-11T05:02:33.562898+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:33.562898+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8818,"log_dups_size":0,"ondisk_log_size":8818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32044192,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1605}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8910","reported_seq":13096,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.579327+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:33.579327+0000","last_peered":"2024-09-11T05:02:33.579327+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:33.579327+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32068768,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8854","reported_seq":8972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:29.978986+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:29.978986+0000","last_peered":"2024-09-11T05:02:29.978986+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:29.978986+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32221504,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8673","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.781836+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:30.781836+0000","last_peered":"2024-09-11T05:02:30.781836+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:30.781836+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31577248,"num_objects":3855,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8671,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8734","reported_seq":8846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.030787+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:30.030787+0000","last_peered":"2024-09-11T05:02:30.030787+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:30.030787+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32100080,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8497","reported_seq":13093,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:29.404349+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:29.404349+0000","last_peered":"2024-09-11T05:02:29.404349+0000","last_clean":"2024-09-11T05:02:29.404349+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:29.404349+0000","last_undegraded":"2024-09-11T05:02:29.404349+0000","last_fullsized":"2024-09-11T05:02:29.404349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8497,"log_dups_size":0,"ondisk_log_size":8497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31842304,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2307,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8782","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.508198+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:30.508198+0000","last_peered":"2024-09-11T05:02:30.508198+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:30.508198+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8782,"log_dups_size":0,"ondisk_log_size":8782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32283216,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1615}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8664","reported_seq":8765,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.064140+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:30.064140+0000","last_peered":"2024-09-11T05:02:30.064140+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:30.064140+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32175264,"num_objects":3928,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1664}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8607","reported_seq":8718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:30.712515+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:30.712515+0000","last_peered":"2024-09-11T05:02:30.712515+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:30.712515+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8607,"log_dups_size":0,"ondisk_log_size":8607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31143072,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8605,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1499}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8845","reported_seq":8960,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.445862+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:33.445862+0000","last_peered":"2024-09-11T05:02:33.445862+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:33.445862+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8843,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":509724296,"num_objects":62228,"num_object_clones":0,"num_object_copies":248912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132965,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1021288448,"data_stored":1021288448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139181,"ondisk_log_size":139181,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719620,"kb_used_data":250608,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93652220,"statfs":{"total":96636764160,"available":95899873280,"internally_reserved":0,"allocated":256622592,"data_stored":256442623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,1,3,2,3,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714460,"kb_used_data":250124,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93657380,"statfs":{"total":96636764160,"available":95905157120,"internally_reserved":0,"allocated":256126976,"data_stored":255947007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576744,"kb_used_data":252164,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93795096,"statfs":{"total":96636764160,"available":96046178304,"internally_reserved":0,"allocated":258215936,"data_stored":258035967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":332328658},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716224,"kb_used_data":247280,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93655616,"statfs":{"total":96636764160,"available":95903350784,"internally_reserved":0,"allocated":253214720,"data_stored":253034751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,3,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252809216,"data_stored":252809216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257228800,"data_stored":257228800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":255238144,"data_stored":255238144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256012288,"data_stored":256012288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:38.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:38.789 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:38.815 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-11T05:02:37.784941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503727752,"num_objects":61496,"num_object_clones":0,"num_object_copies":245984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133697,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139913,"ondisk_log_size":139913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762272,"kb_used_data":986832,"kb_used_omap":14,"kb_used_meta":1775345,"kb_avail":374725088,"statfs":{"total":386547056640,"available":383718490112,"internally_reserved":0,"allocated":1010515968,"data_stored":1009796092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1817953846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,1,5,9,14,15,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20250624,"num_objects":-2472,"num_object_clones":0,"num_object_copies":-9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2472,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006999"},"pg_stats":[{"pgid":"1.f","version":"33'8710","reported_seq":8799,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.891918+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:34.891918+0000","last_peered":"2024-09-11T05:02:34.891918+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:34.891918+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31162368,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8710,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8786","reported_seq":8896,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.583401+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:33.583401+0000","last_peered":"2024-09-11T05:02:33.583401+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:33.583401+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32371872,"num_objects":3952,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8444","reported_seq":8543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.748090+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:35.748090+0000","last_peered":"2024-09-11T05:02:35.748090+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:35.748090+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30440040,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8443,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1506}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8634","reported_seq":8737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.396238+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:33.396238+0000","last_peered":"2024-09-11T05:02:33.396238+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:33.396238+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8634,"log_dups_size":0,"ondisk_log_size":8634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30979232,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8632,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8662","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.682937+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:34.682937+0000","last_peered":"2024-09-11T05:02:34.682937+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:34.682937+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31691936,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8800","reported_seq":8901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.126599+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:35.126599+0000","last_peered":"2024-09-11T05:02:35.126599+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:35.126599+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8800,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8818","reported_seq":8907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.562898+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:33.562898+0000","last_peered":"2024-09-11T05:02:33.562898+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:33.562898+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8818,"log_dups_size":0,"ondisk_log_size":8818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32044192,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1605}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8910","reported_seq":13096,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.579327+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:33.579327+0000","last_peered":"2024-09-11T05:02:33.579327+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:33.579327+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32068768,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8921","reported_seq":9039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.042942+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:35.042942+0000","last_peered":"2024-09-11T05:02:35.042942+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:35.042942+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31672640,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8732","reported_seq":8839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.908074+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:35.908074+0000","last_peered":"2024-09-11T05:02:35.908074+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:35.908074+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8732,"log_dups_size":0,"ondisk_log_size":8732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31093920,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8791","reported_seq":8901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.935089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:34.935089+0000","last_peered":"2024-09-11T05:02:34.935089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:34.935089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8791,"log_dups_size":0,"ondisk_log_size":8791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31633136,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8788,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8576","reported_seq":13172,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:34.538443+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:34.538443+0000","last_peered":"2024-09-11T05:02:34.538443+0000","last_clean":"2024-09-11T05:02:34.538443+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:34.538443+0000","last_undegraded":"2024-09-11T05:02:34.538443+0000","last_fullsized":"2024-09-11T05:02:34.538443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8851","reported_seq":8943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.732333+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:35.732333+0000","last_peered":"2024-09-11T05:02:35.732333+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:35.732333+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8851,"log_dups_size":0,"ondisk_log_size":8851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31717968,"num_objects":3872,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8850,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8749","reported_seq":8850,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.003555+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:35.003555+0000","last_peered":"2024-09-11T05:02:35.003555+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:35.003555+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31478944,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8747,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1579}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8684","reported_seq":8798,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.887177+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:35.887177+0000","last_peered":"2024-09-11T05:02:35.887177+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:35.887177+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8684,"log_dups_size":0,"ondisk_log_size":8684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30512288,"num_objects":3725,"num_object_clones":0,"num_object_copies":14900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8682,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1422}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8845","reported_seq":8960,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.445862+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:33.445862+0000","last_peered":"2024-09-11T05:02:33.445862+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:33.445862+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8843,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":503727752,"num_objects":61496,"num_object_clones":0,"num_object_copies":245984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133697,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1007636480,"data_stored":1007636480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139913,"ondisk_log_size":139913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737436,"kb_used_data":246204,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93634404,"statfs":{"total":96636764160,"available":95881629696,"internally_reserved":0,"allocated":252112896,"data_stored":251932927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":502984400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732304,"kb_used_data":245760,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93639536,"statfs":{"total":96636764160,"available":95886884864,"internally_reserved":0,"allocated":251658240,"data_stored":251478271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,1,3,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576308,"kb_used_data":247588,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93795532,"statfs":{"total":96636764160,"available":96046624768,"internally_reserved":0,"allocated":253530112,"data_stored":253350143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336588498},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716224,"kb_used_data":247280,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93655616,"statfs":{"total":96636764160,"available":95903350784,"internally_reserved":0,"allocated":253214720,"data_stored":253034751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,3,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252809216,"data_stored":252809216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252592128,"data_stored":252592128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250720256,"data_stored":250720256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251514880,"data_stored":251514880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:38.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:39.441 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:39.442 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:39.466 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-11T05:02:37.784941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503727752,"num_objects":61496,"num_object_clones":0,"num_object_copies":245984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133697,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139913,"ondisk_log_size":139913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762272,"kb_used_data":986832,"kb_used_omap":14,"kb_used_meta":1775345,"kb_avail":374725088,"statfs":{"total":386547056640,"available":383718490112,"internally_reserved":0,"allocated":1010515968,"data_stored":1009796092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1817953846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,1,5,9,14,15,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20250624,"num_objects":-2472,"num_object_clones":0,"num_object_copies":-9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2472,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006999"},"pg_stats":[{"pgid":"1.f","version":"33'8710","reported_seq":8799,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.891918+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:34.891918+0000","last_peered":"2024-09-11T05:02:34.891918+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:34.891918+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31162368,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8710,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8786","reported_seq":8896,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.583401+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:33.583401+0000","last_peered":"2024-09-11T05:02:33.583401+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:33.583401+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32371872,"num_objects":3952,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8444","reported_seq":8543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.748090+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:35.748090+0000","last_peered":"2024-09-11T05:02:35.748090+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:35.748090+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30440040,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8443,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1506}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8634","reported_seq":8737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.396238+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:33.396238+0000","last_peered":"2024-09-11T05:02:33.396238+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:33.396238+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8634,"log_dups_size":0,"ondisk_log_size":8634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30979232,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8632,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8662","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.682937+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:34.682937+0000","last_peered":"2024-09-11T05:02:34.682937+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:34.682937+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31691936,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8800","reported_seq":8901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.126599+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:35.126599+0000","last_peered":"2024-09-11T05:02:35.126599+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:35.126599+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8800,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8818","reported_seq":8907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.562898+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:33.562898+0000","last_peered":"2024-09-11T05:02:33.562898+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:33.562898+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8818,"log_dups_size":0,"ondisk_log_size":8818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32044192,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1605}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8910","reported_seq":13096,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.579327+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:33.579327+0000","last_peered":"2024-09-11T05:02:33.579327+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:33.579327+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32068768,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2061,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8921","reported_seq":9039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.042942+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:35.042942+0000","last_peered":"2024-09-11T05:02:35.042942+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:35.042942+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31672640,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8732","reported_seq":8839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.908074+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:35.908074+0000","last_peered":"2024-09-11T05:02:35.908074+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:35.908074+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8732,"log_dups_size":0,"ondisk_log_size":8732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31093920,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8791","reported_seq":8901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.935089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:34.935089+0000","last_peered":"2024-09-11T05:02:34.935089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:34.935089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8791,"log_dups_size":0,"ondisk_log_size":8791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31633136,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8788,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8576","reported_seq":13172,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:34.538443+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:34.538443+0000","last_peered":"2024-09-11T05:02:34.538443+0000","last_clean":"2024-09-11T05:02:34.538443+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:34.538443+0000","last_undegraded":"2024-09-11T05:02:34.538443+0000","last_fullsized":"2024-09-11T05:02:34.538443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8851","reported_seq":8943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.732333+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:35.732333+0000","last_peered":"2024-09-11T05:02:35.732333+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:35.732333+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8851,"log_dups_size":0,"ondisk_log_size":8851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31717968,"num_objects":3872,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8850,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8749","reported_seq":8850,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.003555+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:35.003555+0000","last_peered":"2024-09-11T05:02:35.003555+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:35.003555+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31478944,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8747,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1579}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8684","reported_seq":8798,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.887177+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:35.887177+0000","last_peered":"2024-09-11T05:02:35.887177+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:35.887177+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8684,"log_dups_size":0,"ondisk_log_size":8684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30512288,"num_objects":3725,"num_object_clones":0,"num_object_copies":14900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8682,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1422}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8845","reported_seq":8960,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:33.445862+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:33.445862+0000","last_peered":"2024-09-11T05:02:33.445862+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:33.445862+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208032,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8843,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":503727752,"num_objects":61496,"num_object_clones":0,"num_object_copies":245984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133697,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1007636480,"data_stored":1007636480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139913,"ondisk_log_size":139913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737436,"kb_used_data":246204,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93634404,"statfs":{"total":96636764160,"available":95881629696,"internally_reserved":0,"allocated":252112896,"data_stored":251932927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":502984400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732304,"kb_used_data":245760,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93639536,"statfs":{"total":96636764160,"available":95886884864,"internally_reserved":0,"allocated":251658240,"data_stored":251478271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,1,3,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576308,"kb_used_data":247588,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93795532,"statfs":{"total":96636764160,"available":96046624768,"internally_reserved":0,"allocated":253530112,"data_stored":253350143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336588498},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716224,"kb_used_data":247280,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93655616,"statfs":{"total":96636764160,"available":95903350784,"internally_reserved":0,"allocated":253214720,"data_stored":253034751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,3,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252809216,"data_stored":252809216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252592128,"data_stored":252592128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250720256,"data_stored":250720256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251514880,"data_stored":251514880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:39.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:39.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8710", 'reported_seq': 8799, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:34.891918+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:34.891918+0000', 'last_peered': '2024-09-11T05:02:34.891918+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:34.891918+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8710, 'log_dups_size': 0, 'ondisk_log_size': 8710, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31162368, 'num_objects': 3804, 'num_object_clones': 0, 'num_object_copies': 15216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8710, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1513}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:39.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:39.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8786", 'reported_seq': 8896, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:33.583401+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:33.583401+0000', 'last_peered': '2024-09-11T05:02:33.583401+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:33.583401+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8786, 'log_dups_size': 0, 'ondisk_log_size': 8786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32371872, 'num_objects': 3952, 'num_object_clones': 0, 'num_object_copies': 15808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8784, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1694}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:39.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:39.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8444", 'reported_seq': 8543, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:35.748090+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:35.748090+0000', 'last_peered': '2024-09-11T05:02:35.748090+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:35.748090+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8444, 'log_dups_size': 0, 'ondisk_log_size': 8444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30440040, 'num_objects': 3717, 'num_object_clones': 0, 'num_object_copies': 14868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3717, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8443, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1506}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:39.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:39.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8634", 'reported_seq': 8737, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:33.396238+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:33.396238+0000', 'last_peered': '2024-09-11T05:02:33.396238+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:33.396238+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8634, 'log_dups_size': 0, 'ondisk_log_size': 8634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30979232, 'num_objects': 3782, 'num_object_clones': 0, 'num_object_copies': 15128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8632, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:39.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:39.470 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8662", 'reported_seq': 8780, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:34.682937+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:34.682937+0000', 'last_peered': '2024-09-11T05:02:34.682937+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:34.682937+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8662, 'log_dups_size': 0, 'ondisk_log_size': 8662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31691936, 'num_objects': 3869, 'num_object_clones': 0, 'num_object_copies': 15476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8660, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1648}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:39.470 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:39.470 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8800", 'reported_seq': 8901, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:35.126599+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:35.126599+0000', 'last_peered': '2024-09-11T05:02:35.126599+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:35.126599+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8800, 'log_dups_size': 0, 'ondisk_log_size': 8800, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31457280, 'num_objects': 3840, 'num_object_clones': 0, 'num_object_copies': 15360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8800, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1546}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:39.470 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:39.470 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8818", 'reported_seq': 8907, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:33.562898+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:33.562898+0000', 'last_peered': '2024-09-11T05:02:33.562898+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:33.562898+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8818, 'log_dups_size': 0, 'ondisk_log_size': 8818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32044192, 'num_objects': 3912, 'num_object_clones': 0, 'num_object_copies': 15648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8816, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1605}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:39.471 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:39.471 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8910", 'reported_seq': 13096, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:33.579327+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:33.579327+0000', 'last_peered': '2024-09-11T05:02:33.579327+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:33.579327+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8910, 'log_dups_size': 0, 'ondisk_log_size': 8910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32068768, 'num_objects': 3915, 'num_object_clones': 0, 'num_object_copies': 15660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8908, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2061, 'num_bytes_recovered': 15564800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3611}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:39.471 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:39.471 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8921", 'reported_seq': 9039, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:35.042942+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:35.042942+0000', 'last_peered': '2024-09-11T05:02:35.042942+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:35.042942+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8921, 'log_dups_size': 0, 'ondisk_log_size': 8921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31672640, 'num_objects': 3867, 'num_object_clones': 0, 'num_object_copies': 15468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8918, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1501}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:39.472 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:39.472 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8732", 'reported_seq': 8839, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:35.908074+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:35.908074+0000', 'last_peered': '2024-09-11T05:02:35.908074+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:35.908074+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8732, 'log_dups_size': 0, 'ondisk_log_size': 8732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31093920, 'num_objects': 3796, 'num_object_clones': 0, 'num_object_copies': 15184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8730, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1490}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:39.472 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:39.472 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8791", 'reported_seq': 8901, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:34.935089+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:34.935089+0000', 'last_peered': '2024-09-11T05:02:34.935089+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:34.935089+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8791, 'log_dups_size': 0, 'ondisk_log_size': 8791, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31633136, 'num_objects': 3862, 'num_object_clones': 0, 'num_object_copies': 15448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8788, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1564}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:39.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:39.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8851", 'reported_seq': 8943, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:35.732333+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:35.732333+0000', 'last_peered': '2024-09-11T05:02:35.732333+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:35.732333+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8851, 'log_dups_size': 0, 'ondisk_log_size': 8851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31717968, 'num_objects': 3872, 'num_object_clones': 0, 'num_object_copies': 15488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8850, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1546}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:39.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:39.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8749", 'reported_seq': 8850, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:35.003555+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:35.003555+0000', 'last_peered': '2024-09-11T05:02:35.003555+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:35.003555+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8749, 'log_dups_size': 0, 'ondisk_log_size': 8749, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31478944, 'num_objects': 3843, 'num_object_clones': 0, 'num_object_copies': 15372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8747, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1579}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:39.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:39.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8684", 'reported_seq': 8798, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:35.887177+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:35.887177+0000', 'last_peered': '2024-09-11T05:02:35.887177+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:35.887177+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8684, 'log_dups_size': 0, 'ondisk_log_size': 8684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30512288, 'num_objects': 3725, 'num_object_clones': 0, 'num_object_copies': 14900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8682, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1422}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:39.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:39.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8845", 'reported_seq': 8960, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:33.445862+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:33.445862+0000', 'last_peered': '2024-09-11T05:02:33.445862+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:33.445862+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8845, 'log_dups_size': 0, 'ondisk_log_size': 8845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32208032, 'num_objects': 3932, 'num_object_clones': 0, 'num_object_copies': 15728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8843, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1644}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:39.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:40.249 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:40.249 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32322/245984 objects degraded (13.140%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":487,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61496,"data_bytes":503727752,"bytes_used":2828566528,"bytes_avail":383718490112,"bytes_total":386547056640,"degraded_objects":32322,"degraded_total":245984,"degraded_ratio":0.131398790165214,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":205},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:40.278 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:40.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:40.904 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:40.905 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:40.925 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-11T05:02:39.785642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":500835976,"num_objects":61143,"num_object_clones":0,"num_object_copies":244572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61143,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134050,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2406,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140266,"ondisk_log_size":140266,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775984,"kb_used_data":982492,"kb_used_omap":14,"kb_used_meta":1793393,"kb_avail":374711376,"statfs":{"total":386547056640,"available":383704449024,"internally_reserved":0,"allocated":1006071808,"data_stored":1005351932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1836434998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,3,10,11,13,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20914176,"num_objects":-2553,"num_object_clones":0,"num_object_copies":-10212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006823"},"pg_stats":[{"pgid":"1.f","version":"33'8710","reported_seq":8799,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.891918+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:34.891918+0000","last_peered":"2024-09-11T05:02:34.891918+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:34.891918+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31162368,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8710,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8845","reported_seq":8954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:38.470318+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:38.470318+0000","last_peered":"2024-09-11T05:02:38.470318+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:38.470318+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31888544,"num_objects":3893,"num_object_clones":0,"num_object_copies":15572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8843,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8444","reported_seq":8543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.748090+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:35.748090+0000","last_peered":"2024-09-11T05:02:35.748090+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:35.748090+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30440040,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8443,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1506}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8709","reported_seq":8812,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:38.654295+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:38.654295+0000","last_peered":"2024-09-11T05:02:38.654295+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:38.654295+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30364832,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8662","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.682937+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:34.682937+0000","last_peered":"2024-09-11T05:02:34.682937+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:34.682937+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31691936,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8800","reported_seq":8901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.126599+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:35.126599+0000","last_peered":"2024-09-11T05:02:35.126599+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:35.126599+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":3840,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8800,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8884","reported_seq":8973,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:38.545821+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:38.545821+0000","last_peered":"2024-09-11T05:02:38.545821+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:38.545821+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8884,"log_dups_size":0,"ondisk_log_size":8884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503520,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8882,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1539}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8983","reported_seq":13280,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:38.539336+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:38.539336+0000","last_peered":"2024-09-11T05:02:38.539336+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:38.539336+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8983,"log_dups_size":0,"ondisk_log_size":8983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470752,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2116,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8921","reported_seq":9039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.042942+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:35.042942+0000","last_peered":"2024-09-11T05:02:35.042942+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:35.042942+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31672640,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8732","reported_seq":8839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.908074+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:35.908074+0000","last_peered":"2024-09-11T05:02:35.908074+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:35.908074+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8732,"log_dups_size":0,"ondisk_log_size":8732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31093920,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8730,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8791","reported_seq":8901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:34.935089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:34.935089+0000","last_peered":"2024-09-11T05:02:34.935089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:34.935089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8791,"log_dups_size":0,"ondisk_log_size":8791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31633136,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8788,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8576","reported_seq":13172,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:34.538443+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:34.538443+0000","last_peered":"2024-09-11T05:02:34.538443+0000","last_clean":"2024-09-11T05:02:34.538443+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:34.538443+0000","last_undegraded":"2024-09-11T05:02:34.538443+0000","last_fullsized":"2024-09-11T05:02:34.538443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8851","reported_seq":8943,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.732333+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:35.732333+0000","last_peered":"2024-09-11T05:02:35.732333+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:35.732333+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8851,"log_dups_size":0,"ondisk_log_size":8851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31717968,"num_objects":3872,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8850,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8749","reported_seq":8850,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.003555+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:35.003555+0000","last_peered":"2024-09-11T05:02:35.003555+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:35.003555+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31478944,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8747,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1579}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8684","reported_seq":8798,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:35.887177+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:35.887177+0000","last_peered":"2024-09-11T05:02:35.887177+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:35.887177+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8684,"log_dups_size":0,"ondisk_log_size":8684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30512288,"num_objects":3725,"num_object_clones":0,"num_object_copies":14900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8682,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1422}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8925","reported_seq":9039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:38.415633+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:38.415633+0000","last_peered":"2024-09-11T05:02:38.415633+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:38.415633+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31552672,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8923,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":500835976,"num_objects":61143,"num_object_clones":0,"num_object_copies":244572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61143,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134050,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2406,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1003085824,"data_stored":1003085824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140266,"ondisk_log_size":140266,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737436,"kb_used_data":246204,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93634404,"statfs":{"total":96636764160,"available":95881629696,"internally_reserved":0,"allocated":252112896,"data_stored":251932927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":502984400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732304,"kb_used_data":245760,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93639536,"statfs":{"total":96636764160,"available":95886884864,"internally_reserved":0,"allocated":251658240,"data_stored":251478271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,1,1,3,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576308,"kb_used_data":247588,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93795532,"statfs":{"total":96636764160,"available":96046624768,"internally_reserved":0,"allocated":253530112,"data_stored":253350143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":336588498},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729936,"kb_used_data":242940,"kb_used_omap":4,"kb_used_meta":486971,"kb_avail":93641904,"statfs":{"total":96636764160,"available":95889309696,"internally_reserved":0,"allocated":248770560,"data_stored":248590591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":498659023},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":248258560,"data_stored":248258560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252592128,"data_stored":252592128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250720256,"data_stored":250720256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251514880,"data_stored":251514880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:43.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:44.519 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:44.541 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-11T05:02:43.788678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":491972232,"num_objects":60061,"num_object_clones":0,"num_object_copies":240244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135132,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2467,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141348,"ondisk_log_size":141348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742004,"kb_used_data":965164,"kb_used_omap":14,"kb_used_meta":1776753,"kb_avail":374745356,"statfs":{"total":386547056640,"available":383739244544,"internally_reserved":0,"allocated":988327936,"data_stored":987608060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1819395638},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,16,16,5,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20783104,"num_objects":-2537,"num_object_clones":0,"num_object_copies":-10148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2537,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007767"},"pg_stats":[{"pgid":"1.f","version":"33'8784","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.123294+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:40.123294+0000","last_peered":"2024-09-11T05:02:40.123294+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:40.123294+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8784,"log_dups_size":0,"ondisk_log_size":8784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30556160,"num_objects":3730,"num_object_clones":0,"num_object_copies":14920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8922","reported_seq":9032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.732790+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:43.732790+0000","last_peered":"2024-09-11T05:02:43.732790+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:43.732790+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31257760,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8507","reported_seq":8607,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.800227+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:40.800227+0000","last_peered":"2024-09-11T05:02:40.800227+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:40.800227+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8507,"log_dups_size":0,"ondisk_log_size":8507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29923944,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8506,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1443}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8772","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.683064+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:43.683064+0000","last_peered":"2024-09-11T05:02:43.683064+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:43.683064+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8772,"log_dups_size":0,"ondisk_log_size":8772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29848736,"num_objects":3644,"num_object_clones":0,"num_object_copies":14576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8740","reported_seq":8858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:39.919571+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:39.919571+0000","last_peered":"2024-09-11T05:02:39.919571+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:39.919571+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31052960,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8860","reported_seq":8961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.049235+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:40.049235+0000","last_peered":"2024-09-11T05:02:40.049235+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:40.049235+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8860,"log_dups_size":0,"ondisk_log_size":8860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30965760,"num_objects":3780,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8860,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8953","reported_seq":9042,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.592640+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:43.592640+0000","last_peered":"2024-09-11T05:02:43.592640+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:43.592640+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8953,"log_dups_size":0,"ondisk_log_size":8953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30938272,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8951,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1470}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9053","reported_seq":13472,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.726471+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:43.726471+0000","last_peered":"2024-09-11T05:02:43.726471+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:43.726471+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30897312,"num_objects":3772,"num_object_clones":0,"num_object_copies":15088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8985","reported_seq":9103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.134838+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:40.134838+0000","last_peered":"2024-09-11T05:02:40.134838+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:40.134838+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31148352,"num_objects":3803,"num_object_clones":0,"num_object_copies":15212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8982,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8807","reported_seq":8914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.749137+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:40.749137+0000","last_peered":"2024-09-11T05:02:40.749137+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:40.749137+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30479520,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8805,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1415}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8858","reported_seq":8970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.253988+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:40.253988+0000","last_peered":"2024-09-11T05:02:40.253988+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:40.253988+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8858,"log_dups_size":0,"ondisk_log_size":8858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31084272,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8855,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8636","reported_seq":13232,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:39.804145+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:39.804145+0000","last_peered":"2024-09-11T05:02:39.804145+0000","last_clean":"2024-09-11T05:02:39.804145+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:39.804145+0000","last_undegraded":"2024-09-11T05:02:39.804145+0000","last_fullsized":"2024-09-11T05:02:39.804145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30703616,"num_objects":3748,"num_object_clones":0,"num_object_copies":14992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2446,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8908","reported_seq":9000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.842138+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:40.842138+0000","last_peered":"2024-09-11T05:02:40.842138+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:40.842138+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8908,"log_dups_size":0,"ondisk_log_size":8908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31251024,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8834","reported_seq":8936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.180770+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:40.180770+0000","last_peered":"2024-09-11T05:02:40.180770+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:40.180770+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8834,"log_dups_size":0,"ondisk_log_size":8834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30782624,"num_objects":3758,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8832,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8746","reported_seq":8860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.738734+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:40.738734+0000","last_peered":"2024-09-11T05:02:40.738734+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:40.738734+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8746,"log_dups_size":0,"ondisk_log_size":8746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30004384,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8983","reported_seq":9098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.660094+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:43.660094+0000","last_peered":"2024-09-11T05:02:43.660094+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:43.660094+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8983,"log_dups_size":0,"ondisk_log_size":8983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31077536,"num_objects":3794,"num_object_clones":0,"num_object_copies":15176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":491972232,"num_objects":60061,"num_object_clones":0,"num_object_copies":240244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135132,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2467,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":985227264,"data_stored":985227264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141348,"ondisk_log_size":141348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720616,"kb_used_data":241880,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93651224,"statfs":{"total":96636764160,"available":95898853376,"internally_reserved":0,"allocated":247685120,"data_stored":247505151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":490204880},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732132,"kb_used_data":241424,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93639708,"statfs":{"total":96636764160,"available":95887060992,"internally_reserved":0,"allocated":247218176,"data_stored":247038207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559436,"kb_used_data":243196,"kb_used_omap":4,"kb_used_meta":316219,"kb_avail":93812404,"statfs":{"total":96636764160,"available":96063901696,"internally_reserved":0,"allocated":249032704,"data_stored":248852735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323808978},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729820,"kb_used_data":238664,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93642020,"statfs":{"total":96636764160,"available":95889428480,"internally_reserved":0,"allocated":244391936,"data_stored":244211967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502918863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,5,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243884032,"data_stored":243884032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248012800,"data_stored":248012800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246378496,"data_stored":246378496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":246951936,"data_stored":246951936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:44.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:45.086 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:45.103 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-11T05:02:43.788678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":491972232,"num_objects":60061,"num_object_clones":0,"num_object_copies":240244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135132,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2467,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141348,"ondisk_log_size":141348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742004,"kb_used_data":965164,"kb_used_omap":14,"kb_used_meta":1776753,"kb_avail":374745356,"statfs":{"total":386547056640,"available":383739244544,"internally_reserved":0,"allocated":988327936,"data_stored":987608060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1819395638},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,16,16,5,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20783104,"num_objects":-2537,"num_object_clones":0,"num_object_copies":-10148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2537,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007767"},"pg_stats":[{"pgid":"1.f","version":"33'8784","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.123294+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:40.123294+0000","last_peered":"2024-09-11T05:02:40.123294+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:40.123294+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8784,"log_dups_size":0,"ondisk_log_size":8784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30556160,"num_objects":3730,"num_object_clones":0,"num_object_copies":14920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8784,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8922","reported_seq":9032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.732790+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:43.732790+0000","last_peered":"2024-09-11T05:02:43.732790+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:43.732790+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31257760,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8507","reported_seq":8607,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.800227+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:40.800227+0000","last_peered":"2024-09-11T05:02:40.800227+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:40.800227+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8507,"log_dups_size":0,"ondisk_log_size":8507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29923944,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8506,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1443}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8772","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.683064+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:43.683064+0000","last_peered":"2024-09-11T05:02:43.683064+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:43.683064+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8772,"log_dups_size":0,"ondisk_log_size":8772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29848736,"num_objects":3644,"num_object_clones":0,"num_object_copies":14576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8740","reported_seq":8858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:39.919571+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:39.919571+0000","last_peered":"2024-09-11T05:02:39.919571+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:39.919571+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31052960,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8860","reported_seq":8961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.049235+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:40.049235+0000","last_peered":"2024-09-11T05:02:40.049235+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:40.049235+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8860,"log_dups_size":0,"ondisk_log_size":8860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30965760,"num_objects":3780,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8860,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8953","reported_seq":9042,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.592640+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:43.592640+0000","last_peered":"2024-09-11T05:02:43.592640+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:43.592640+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8953,"log_dups_size":0,"ondisk_log_size":8953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30938272,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8951,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1470}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9053","reported_seq":13472,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.726471+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:43.726471+0000","last_peered":"2024-09-11T05:02:43.726471+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:43.726471+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30897312,"num_objects":3772,"num_object_clones":0,"num_object_copies":15088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8985","reported_seq":9103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.134838+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:40.134838+0000","last_peered":"2024-09-11T05:02:40.134838+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:40.134838+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31148352,"num_objects":3803,"num_object_clones":0,"num_object_copies":15212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8982,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8807","reported_seq":8914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.749137+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:40.749137+0000","last_peered":"2024-09-11T05:02:40.749137+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:40.749137+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30479520,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8805,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1415}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8858","reported_seq":8970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.253988+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:40.253988+0000","last_peered":"2024-09-11T05:02:40.253988+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:40.253988+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8858,"log_dups_size":0,"ondisk_log_size":8858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31084272,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8855,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8636","reported_seq":13232,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:39.804145+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:39.804145+0000","last_peered":"2024-09-11T05:02:39.804145+0000","last_clean":"2024-09-11T05:02:39.804145+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:39.804145+0000","last_undegraded":"2024-09-11T05:02:39.804145+0000","last_fullsized":"2024-09-11T05:02:39.804145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8636,"log_dups_size":0,"ondisk_log_size":8636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30703616,"num_objects":3748,"num_object_clones":0,"num_object_copies":14992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2446,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8908","reported_seq":9000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.842138+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:40.842138+0000","last_peered":"2024-09-11T05:02:40.842138+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:40.842138+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8908,"log_dups_size":0,"ondisk_log_size":8908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31251024,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8834","reported_seq":8936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.180770+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:40.180770+0000","last_peered":"2024-09-11T05:02:40.180770+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:40.180770+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8834,"log_dups_size":0,"ondisk_log_size":8834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30782624,"num_objects":3758,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8832,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8746","reported_seq":8860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.738734+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:40.738734+0000","last_peered":"2024-09-11T05:02:40.738734+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:40.738734+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8746,"log_dups_size":0,"ondisk_log_size":8746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30004384,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8983","reported_seq":9098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.660094+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:43.660094+0000","last_peered":"2024-09-11T05:02:43.660094+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:43.660094+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8983,"log_dups_size":0,"ondisk_log_size":8983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31077536,"num_objects":3794,"num_object_clones":0,"num_object_copies":15176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":491972232,"num_objects":60061,"num_object_clones":0,"num_object_copies":240244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135132,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2467,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":985227264,"data_stored":985227264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141348,"ondisk_log_size":141348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720616,"kb_used_data":241880,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93651224,"statfs":{"total":96636764160,"available":95898853376,"internally_reserved":0,"allocated":247685120,"data_stored":247505151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":490204880},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732132,"kb_used_data":241424,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93639708,"statfs":{"total":96636764160,"available":95887060992,"internally_reserved":0,"allocated":247218176,"data_stored":247038207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559436,"kb_used_data":243196,"kb_used_omap":4,"kb_used_meta":316219,"kb_avail":93812404,"statfs":{"total":96636764160,"available":96063901696,"internally_reserved":0,"allocated":249032704,"data_stored":248852735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":323808978},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729820,"kb_used_data":238664,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93642020,"statfs":{"total":96636764160,"available":95889428480,"internally_reserved":0,"allocated":244391936,"data_stored":244211967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502918863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,5,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243884032,"data_stored":243884032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248012800,"data_stored":248012800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246378496,"data_stored":246378496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":246951936,"data_stored":246951936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:45.104 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:45.104 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8784", 'reported_seq': 8874, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.123294+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:40.123294+0000', 'last_peered': '2024-09-11T05:02:40.123294+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:40.123294+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8784, 'log_dups_size': 0, 'ondisk_log_size': 8784, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30556160, 'num_objects': 3730, 'num_object_clones': 0, 'num_object_copies': 14920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8784, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1439}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:45.104 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:45.104 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8922", 'reported_seq': 9032, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:43.732790+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:43.732790+0000', 'last_peered': '2024-09-11T05:02:43.732790+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:43.732790+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8922, 'log_dups_size': 0, 'ondisk_log_size': 8922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31257760, 'num_objects': 3816, 'num_object_clones': 0, 'num_object_copies': 15264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8920, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1558}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:45.104 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8507", 'reported_seq': 8607, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.800227+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:40.800227+0000', 'last_peered': '2024-09-11T05:02:40.800227+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:40.800227+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8507, 'log_dups_size': 0, 'ondisk_log_size': 8507, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29923944, 'num_objects': 3654, 'num_object_clones': 0, 'num_object_copies': 14616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3654, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8506, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1443}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8772", 'reported_seq': 8874, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:43.683064+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:43.683064+0000', 'last_peered': '2024-09-11T05:02:43.683064+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:43.683064+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8772, 'log_dups_size': 0, 'ondisk_log_size': 8772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29848736, 'num_objects': 3644, 'num_object_clones': 0, 'num_object_copies': 14576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8770, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8740", 'reported_seq': 8858, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:39.919571+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:39.919571+0000', 'last_peered': '2024-09-11T05:02:39.919571+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:39.919571+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8740, 'log_dups_size': 0, 'ondisk_log_size': 8740, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31052960, 'num_objects': 3791, 'num_object_clones': 0, 'num_object_copies': 15164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8738, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1570}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8860", 'reported_seq': 8961, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.049235+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:40.049235+0000', 'last_peered': '2024-09-11T05:02:40.049235+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:40.049235+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8860, 'log_dups_size': 0, 'ondisk_log_size': 8860, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30965760, 'num_objects': 3780, 'num_object_clones': 0, 'num_object_copies': 15120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8860, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8953", 'reported_seq': 9042, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:43.592640+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:43.592640+0000', 'last_peered': '2024-09-11T05:02:43.592640+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:43.592640+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8953, 'log_dups_size': 0, 'ondisk_log_size': 8953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30938272, 'num_objects': 3777, 'num_object_clones': 0, 'num_object_copies': 15108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8951, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1470}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:45.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9053", 'reported_seq': 13472, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:43.726471+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:43.726471+0000', 'last_peered': '2024-09-11T05:02:43.726471+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:43.726471+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9053, 'log_dups_size': 0, 'ondisk_log_size': 9053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30897312, 'num_objects': 3772, 'num_object_clones': 0, 'num_object_copies': 15088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 188, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9051, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2177, 'num_bytes_recovered': 15564800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3584}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8985", 'reported_seq': 9103, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.134838+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:40.134838+0000', 'last_peered': '2024-09-11T05:02:40.134838+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:40.134838+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8985, 'log_dups_size': 0, 'ondisk_log_size': 8985, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31148352, 'num_objects': 3803, 'num_object_clones': 0, 'num_object_copies': 15212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8982, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1437}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8807", 'reported_seq': 8914, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.749137+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:40.749137+0000', 'last_peered': '2024-09-11T05:02:40.749137+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:40.749137+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8807, 'log_dups_size': 0, 'ondisk_log_size': 8807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30479520, 'num_objects': 3721, 'num_object_clones': 0, 'num_object_copies': 14884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8805, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1415}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8858", 'reported_seq': 8970, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.253988+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:40.253988+0000', 'last_peered': '2024-09-11T05:02:40.253988+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:40.253988+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8858, 'log_dups_size': 0, 'ondisk_log_size': 8858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31084272, 'num_objects': 3795, 'num_object_clones': 0, 'num_object_copies': 15180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8855, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8908", 'reported_seq': 9000, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.842138+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:40.842138+0000', 'last_peered': '2024-09-11T05:02:40.842138+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:40.842138+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8908, 'log_dups_size': 0, 'ondisk_log_size': 8908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31251024, 'num_objects': 3815, 'num_object_clones': 0, 'num_object_copies': 15260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8907, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:45.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8834", 'reported_seq': 8936, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.180770+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:40.180770+0000', 'last_peered': '2024-09-11T05:02:40.180770+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:40.180770+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8834, 'log_dups_size': 0, 'ondisk_log_size': 8834, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30782624, 'num_objects': 3758, 'num_object_clones': 0, 'num_object_copies': 15032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8832, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:45.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:45.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8746", 'reported_seq': 8860, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:40.738734+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:40.738734+0000', 'last_peered': '2024-09-11T05:02:40.738734+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:40.738734+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8746, 'log_dups_size': 0, 'ondisk_log_size': 8746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30004384, 'num_objects': 3663, 'num_object_clones': 0, 'num_object_copies': 14652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8744, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1360}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:45.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:45.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8983", 'reported_seq': 9098, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:43.660094+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:43.660094+0000', 'last_peered': '2024-09-11T05:02:43.660094+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:43.660094+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8983, 'log_dups_size': 0, 'ondisk_log_size': 8983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31077536, 'num_objects': 3794, 'num_object_clones': 0, 'num_object_copies': 15176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8981, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1506}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:45.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:45.847 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:45.848 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32206/240244 objects degraded (13.406%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":492,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":60061,"data_bytes":491972232,"bytes_used":2807812096,"bytes_avail":383739244544,"bytes_total":386547056640,"degraded_objects":32206,"degraded_total":240244,"degraded_ratio":0.13405537703334944,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":211},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:45.883 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:45.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:46.556 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:46.574 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-11T05:02:45.789522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":487982728,"num_objects":59574,"num_object_clones":0,"num_object_copies":238296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59574,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135619,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2467,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141835,"ondisk_log_size":141835,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777888,"kb_used_data":956624,"kb_used_omap":14,"kb_used_meta":1821169,"kb_avail":374709472,"statfs":{"total":386547056640,"available":383702499328,"internally_reserved":0,"allocated":979582976,"data_stored":978863100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1864877622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,9,14,5,8,9,13,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21741568,"num_objects":-2654,"num_object_clones":0,"num_object_copies":-10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007040"},"pg_stats":[{"pgid":"1.f","version":"33'8849","reported_seq":8939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.115236+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:45.115236+0000","last_peered":"2024-09-11T05:02:45.115236+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:45.115236+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8849,"log_dups_size":0,"ondisk_log_size":8849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8849,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1374}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8922","reported_seq":9032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.732790+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:43.732790+0000","last_peered":"2024-09-11T05:02:43.732790+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:43.732790+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31257760,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8507","reported_seq":8607,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.800227+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:40.800227+0000","last_peered":"2024-09-11T05:02:40.800227+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:40.800227+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8507,"log_dups_size":0,"ondisk_log_size":8507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29923944,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8506,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1443}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8772","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.683064+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:43.683064+0000","last_peered":"2024-09-11T05:02:43.683064+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:43.683064+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8772,"log_dups_size":0,"ondisk_log_size":8772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29848736,"num_objects":3644,"num_object_clones":0,"num_object_copies":14576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8813","reported_seq":8929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.292870+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:45.292870+0000","last_peered":"2024-09-11T05:02:45.292870+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:45.292870+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8813,"log_dups_size":0,"ondisk_log_size":8813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30454944,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8811,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8933","reported_seq":9034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.270503+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:45.270503+0000","last_peered":"2024-09-11T05:02:45.270503+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:45.270503+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8933,"log_dups_size":0,"ondisk_log_size":8933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30367744,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8933,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8953","reported_seq":9042,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.592640+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:43.592640+0000","last_peered":"2024-09-11T05:02:43.592640+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:43.592640+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8953,"log_dups_size":0,"ondisk_log_size":8953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30938272,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8951,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1470}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9053","reported_seq":13472,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.726471+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:43.726471+0000","last_peered":"2024-09-11T05:02:43.726471+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:43.726471+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30897312,"num_objects":3772,"num_object_clones":0,"num_object_copies":15088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9069","reported_seq":9187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.270758+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:45.270758+0000","last_peered":"2024-09-11T05:02:45.270758+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:45.270758+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30460224,"num_objects":3719,"num_object_clones":0,"num_object_copies":14876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8807","reported_seq":8914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.749137+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:40.749137+0000","last_peered":"2024-09-11T05:02:40.749137+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:40.749137+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8807,"log_dups_size":0,"ondisk_log_size":8807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30479520,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8805,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1415}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8921","reported_seq":9033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.233757+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:45.233757+0000","last_peered":"2024-09-11T05:02:45.233757+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:45.233757+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30568176,"num_objects":3732,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8698","reported_seq":13294,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:44.407435+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:44.407435+0000","last_peered":"2024-09-11T05:02:44.407435+0000","last_clean":"2024-09-11T05:02:44.407435+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:44.407435+0000","last_undegraded":"2024-09-11T05:02:44.407435+0000","last_fullsized":"2024-09-11T05:02:44.407435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8698,"log_dups_size":0,"ondisk_log_size":8698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30195712,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8908","reported_seq":9000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.842138+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:40.842138+0000","last_peered":"2024-09-11T05:02:40.842138+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:40.842138+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8908,"log_dups_size":0,"ondisk_log_size":8908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31251024,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8901","reported_seq":9003,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.202484+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:45.202484+0000","last_peered":"2024-09-11T05:02:45.202484+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:45.202484+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30233760,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1427}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8746","reported_seq":8860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:40.738734+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:40.738734+0000","last_peered":"2024-09-11T05:02:40.738734+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:40.738734+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8746,"log_dups_size":0,"ondisk_log_size":8746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30004384,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8983","reported_seq":9098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:43.660094+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:43.660094+0000","last_peered":"2024-09-11T05:02:43.660094+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:43.660094+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8983,"log_dups_size":0,"ondisk_log_size":8983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31077536,"num_objects":3794,"num_object_clones":0,"num_object_copies":15176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":487982728,"num_objects":59574,"num_object_clones":0,"num_object_copies":238296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59574,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135619,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2467,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":976523264,"data_stored":976523264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141835,"ondisk_log_size":141835,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720460,"kb_used_data":237556,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93651380,"statfs":{"total":96636764160,"available":95899013120,"internally_reserved":0,"allocated":243257344,"data_stored":243077375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":494464720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,13,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732132,"kb_used_data":241424,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93639708,"statfs":{"total":96636764160,"available":95887060992,"internally_reserved":0,"allocated":247218176,"data_stored":247038207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595476,"kb_used_data":238980,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93776364,"statfs":{"total":96636764160,"available":96026996736,"internally_reserved":0,"allocated":244715520,"data_stored":244535551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":365031122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729820,"kb_used_data":238664,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93642020,"statfs":{"total":96636764160,"available":95889428480,"internally_reserved":0,"allocated":244391936,"data_stored":244211967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502918863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,5,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243884032,"data_stored":243884032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243621888,"data_stored":243621888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246378496,"data_stored":246378496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242638848,"data_stored":242638848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:49.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:50.269 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:50.292 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-11T05:02:49.791375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483141256,"num_objects":58983,"num_object_clones":0,"num_object_copies":235932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136210,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2557,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142426,"ondisk_log_size":142426,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2780692,"kb_used_data":948300,"kb_used_omap":14,"kb_used_meta":1832305,"kb_avail":374706668,"statfs":{"total":386547056640,"available":383699628032,"internally_reserved":0,"allocated":971059200,"data_stored":970339324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1876280886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,8,11,9,6,15,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20586496,"num_objects":-2513,"num_object_clones":0,"num_object_copies":-10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2513,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006434"},"pg_stats":[{"pgid":"1.f","version":"33'8849","reported_seq":8939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.115236+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:45.115236+0000","last_peered":"2024-09-11T05:02:45.115236+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:45.115236+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8849,"log_dups_size":0,"ondisk_log_size":8849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8849,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1374}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8986","reported_seq":9094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.842346+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:48.842346+0000","last_peered":"2024-09-11T05:02:48.842346+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:48.842346+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30733472,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8984,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8563","reported_seq":8663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.901834+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:45.901834+0000","last_peered":"2024-09-11T05:02:45.901834+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:45.901834+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8563,"log_dups_size":0,"ondisk_log_size":8563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29465192,"num_objects":3598,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8562,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1387}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8829","reported_seq":8932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.578365+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:48.578365+0000","last_peered":"2024-09-11T05:02:48.578365+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:48.578365+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381792,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8813","reported_seq":8929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.292870+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:45.292870+0000","last_peered":"2024-09-11T05:02:45.292870+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:45.292870+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8813,"log_dups_size":0,"ondisk_log_size":8813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30454944,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8811,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8933","reported_seq":9034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.270503+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:45.270503+0000","last_peered":"2024-09-11T05:02:45.270503+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:45.270503+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8933,"log_dups_size":0,"ondisk_log_size":8933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30367744,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8933,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9028","reported_seq":9117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.771909+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:48.771909+0000","last_peered":"2024-09-11T05:02:48.771909+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:48.771909+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9028,"log_dups_size":0,"ondisk_log_size":9028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30323872,"num_objects":3702,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9026,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9128","reported_seq":13726,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.874917+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:48.874917+0000","last_peered":"2024-09-11T05:02:48.874917+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:48.874917+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282912,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9069","reported_seq":9187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.270758+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:45.270758+0000","last_peered":"2024-09-11T05:02:45.270758+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:45.270758+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30460224,"num_objects":3719,"num_object_clones":0,"num_object_copies":14876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8865","reported_seq":8972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.907852+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:45.907852+0000","last_peered":"2024-09-11T05:02:45.907852+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:45.907852+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30004384,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8921","reported_seq":9033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.233757+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:45.233757+0000","last_peered":"2024-09-11T05:02:45.233757+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:45.233757+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30568176,"num_objects":3732,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8698","reported_seq":13294,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:44.407435+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:44.407435+0000","last_peered":"2024-09-11T05:02:44.407435+0000","last_clean":"2024-09-11T05:02:44.407435+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:44.407435+0000","last_undegraded":"2024-09-11T05:02:44.407435+0000","last_fullsized":"2024-09-11T05:02:44.407435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8698,"log_dups_size":0,"ondisk_log_size":8698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30195712,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8976","reported_seq":9068,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.752065+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:45.752065+0000","last_peered":"2024-09-11T05:02:45.752065+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:45.752065+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30693968,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8901","reported_seq":9003,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.202484+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:45.202484+0000","last_peered":"2024-09-11T05:02:45.202484+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:45.202484+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30233760,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1427}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8810","reported_seq":8924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.844624+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:45.844624+0000","last_peered":"2024-09-11T05:02:45.844624+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:45.844624+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8810,"log_dups_size":0,"ondisk_log_size":8810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29480096,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8808,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9057","reported_seq":9172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.653427+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:48.653427+0000","last_peered":"2024-09-11T05:02:48.653427+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:48.653427+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9055,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":483141256,"num_objects":58983,"num_object_clones":0,"num_object_copies":235932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136210,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2557,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":967786496,"data_stored":967786496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142426,"ondisk_log_size":142426,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720460,"kb_used_data":237556,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93651380,"statfs":{"total":96636764160,"available":95899013120,"internally_reserved":0,"allocated":243257344,"data_stored":243077375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":494464720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,13,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733552,"kb_used_data":237280,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93638288,"statfs":{"total":96636764160,"available":95885606912,"internally_reserved":0,"allocated":242974720,"data_stored":242794751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595476,"kb_used_data":238980,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93776364,"statfs":{"total":96636764160,"available":96026996736,"internally_reserved":0,"allocated":244715520,"data_stored":244535551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":365031122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731204,"kb_used_data":234484,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93640636,"statfs":{"total":96636764160,"available":95888011264,"internally_reserved":0,"allocated":240111616,"data_stored":239931647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,8,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239431680,"data_stored":239431680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243621888,"data_stored":243621888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242094080,"data_stored":242094080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242638848,"data_stored":242638848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:50.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:50.946 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:50.947 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:50.974 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-11T05:02:49.791375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483141256,"num_objects":58983,"num_object_clones":0,"num_object_copies":235932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136210,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2557,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142426,"ondisk_log_size":142426,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2780692,"kb_used_data":948300,"kb_used_omap":14,"kb_used_meta":1832305,"kb_avail":374706668,"statfs":{"total":386547056640,"available":383699628032,"internally_reserved":0,"allocated":971059200,"data_stored":970339324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1876280886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,8,11,9,6,15,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20586496,"num_objects":-2513,"num_object_clones":0,"num_object_copies":-10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2513,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006434"},"pg_stats":[{"pgid":"1.f","version":"33'8849","reported_seq":8939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.115236+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:45.115236+0000","last_peered":"2024-09-11T05:02:45.115236+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:45.115236+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8849,"log_dups_size":0,"ondisk_log_size":8849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30023680,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8849,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1374}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8986","reported_seq":9094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.842346+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:48.842346+0000","last_peered":"2024-09-11T05:02:48.842346+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:48.842346+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30733472,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8984,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8563","reported_seq":8663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.901834+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:45.901834+0000","last_peered":"2024-09-11T05:02:45.901834+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:45.901834+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8563,"log_dups_size":0,"ondisk_log_size":8563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29465192,"num_objects":3598,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8562,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1387}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8829","reported_seq":8932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.578365+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:48.578365+0000","last_peered":"2024-09-11T05:02:48.578365+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:48.578365+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381792,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8813","reported_seq":8929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.292870+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:45.292870+0000","last_peered":"2024-09-11T05:02:45.292870+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:45.292870+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8813,"log_dups_size":0,"ondisk_log_size":8813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30454944,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8811,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8933","reported_seq":9034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.270503+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:45.270503+0000","last_peered":"2024-09-11T05:02:45.270503+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:45.270503+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8933,"log_dups_size":0,"ondisk_log_size":8933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30367744,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8933,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9028","reported_seq":9117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.771909+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:48.771909+0000","last_peered":"2024-09-11T05:02:48.771909+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:48.771909+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9028,"log_dups_size":0,"ondisk_log_size":9028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30323872,"num_objects":3702,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9026,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9128","reported_seq":13726,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.874917+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:48.874917+0000","last_peered":"2024-09-11T05:02:48.874917+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:48.874917+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282912,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9069","reported_seq":9187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.270758+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:45.270758+0000","last_peered":"2024-09-11T05:02:45.270758+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:45.270758+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30460224,"num_objects":3719,"num_object_clones":0,"num_object_copies":14876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8865","reported_seq":8972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.907852+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:45.907852+0000","last_peered":"2024-09-11T05:02:45.907852+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:45.907852+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30004384,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8921","reported_seq":9033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.233757+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:45.233757+0000","last_peered":"2024-09-11T05:02:45.233757+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:45.233757+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30568176,"num_objects":3732,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8698","reported_seq":13294,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:44.407435+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:44.407435+0000","last_peered":"2024-09-11T05:02:44.407435+0000","last_clean":"2024-09-11T05:02:44.407435+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:44.407435+0000","last_undegraded":"2024-09-11T05:02:44.407435+0000","last_fullsized":"2024-09-11T05:02:44.407435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8698,"log_dups_size":0,"ondisk_log_size":8698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30195712,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8976","reported_seq":9068,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.752065+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:45.752065+0000","last_peered":"2024-09-11T05:02:45.752065+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:45.752065+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30693968,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8901","reported_seq":9003,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.202484+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:45.202484+0000","last_peered":"2024-09-11T05:02:45.202484+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:45.202484+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30233760,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1427}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8810","reported_seq":8924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:45.844624+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:45.844624+0000","last_peered":"2024-09-11T05:02:45.844624+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:45.844624+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8810,"log_dups_size":0,"ondisk_log_size":8810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29480096,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8808,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9057","reported_seq":9172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.653427+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:48.653427+0000","last_peered":"2024-09-11T05:02:48.653427+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:48.653427+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9055,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":483141256,"num_objects":58983,"num_object_clones":0,"num_object_copies":235932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136210,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2557,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":967786496,"data_stored":967786496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142426,"ondisk_log_size":142426,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720460,"kb_used_data":237556,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93651380,"statfs":{"total":96636764160,"available":95899013120,"internally_reserved":0,"allocated":243257344,"data_stored":243077375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":494464720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,13,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733552,"kb_used_data":237280,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93638288,"statfs":{"total":96636764160,"available":95885606912,"internally_reserved":0,"allocated":242974720,"data_stored":242794751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595476,"kb_used_data":238980,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93776364,"statfs":{"total":96636764160,"available":96026996736,"internally_reserved":0,"allocated":244715520,"data_stored":244535551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":365031122},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731204,"kb_used_data":234484,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93640636,"statfs":{"total":96636764160,"available":95888011264,"internally_reserved":0,"allocated":240111616,"data_stored":239931647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,8,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239431680,"data_stored":239431680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243621888,"data_stored":243621888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242094080,"data_stored":242094080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242638848,"data_stored":242638848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:50.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:50.975 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8849", 'reported_seq': 8939, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.115236+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:45.115236+0000', 'last_peered': '2024-09-11T05:02:45.115236+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:45.115236+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8849, 'log_dups_size': 0, 'ondisk_log_size': 8849, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30023680, 'num_objects': 3665, 'num_object_clones': 0, 'num_object_copies': 14660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8849, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1374}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:50.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8986", 'reported_seq': 9094, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:48.842346+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:48.842346+0000', 'last_peered': '2024-09-11T05:02:48.842346+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:48.842346+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8986, 'log_dups_size': 0, 'ondisk_log_size': 8986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30733472, 'num_objects': 3752, 'num_object_clones': 0, 'num_object_copies': 15008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8984, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8563", 'reported_seq': 8663, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.901834+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:45.901834+0000', 'last_peered': '2024-09-11T05:02:45.901834+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:45.901834+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8563, 'log_dups_size': 0, 'ondisk_log_size': 8563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29465192, 'num_objects': 3598, 'num_object_clones': 0, 'num_object_copies': 14392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3598, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8562, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1387}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8829", 'reported_seq': 8932, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:48.578365+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:48.578365+0000', 'last_peered': '2024-09-11T05:02:48.578365+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:48.578365+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8829, 'log_dups_size': 0, 'ondisk_log_size': 8829, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29381792, 'num_objects': 3587, 'num_object_clones': 0, 'num_object_copies': 14348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8827, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1326}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8813", 'reported_seq': 8929, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.292870+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:45.292870+0000', 'last_peered': '2024-09-11T05:02:45.292870+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:45.292870+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8813, 'log_dups_size': 0, 'ondisk_log_size': 8813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30454944, 'num_objects': 3718, 'num_object_clones': 0, 'num_object_copies': 14872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8811, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8933", 'reported_seq': 9034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.270503+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:45.270503+0000', 'last_peered': '2024-09-11T05:02:45.270503+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:45.270503+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8933, 'log_dups_size': 0, 'ondisk_log_size': 8933, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30367744, 'num_objects': 3707, 'num_object_clones': 0, 'num_object_copies': 14828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8933, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1413}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:50.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'9028", 'reported_seq': 9117, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:48.771909+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:48.771909+0000', 'last_peered': '2024-09-11T05:02:48.771909+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:48.771909+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9028, 'log_dups_size': 0, 'ondisk_log_size': 9028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30323872, 'num_objects': 3702, 'num_object_clones': 0, 'num_object_copies': 14808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9026, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1395}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9128", 'reported_seq': 13726, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:48.874917+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:48.874917+0000', 'last_peered': '2024-09-11T05:02:48.874917+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:48.874917+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9128, 'log_dups_size': 0, 'ondisk_log_size': 9128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30282912, 'num_objects': 3697, 'num_object_clones': 0, 'num_object_copies': 14788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 98, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9126, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2267, 'num_bytes_recovered': 15564800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3599}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'9069", 'reported_seq': 9187, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.270758+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:45.270758+0000', 'last_peered': '2024-09-11T05:02:45.270758+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:45.270758+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9069, 'log_dups_size': 0, 'ondisk_log_size': 9069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30460224, 'num_objects': 3719, 'num_object_clones': 0, 'num_object_copies': 14876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9066, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1353}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8865", 'reported_seq': 8972, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.907852+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:45.907852+0000', 'last_peered': '2024-09-11T05:02:45.907852+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:45.907852+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8865, 'log_dups_size': 0, 'ondisk_log_size': 8865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30004384, 'num_objects': 3663, 'num_object_clones': 0, 'num_object_copies': 14652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8863, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1357}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8921", 'reported_seq': 9033, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.233757+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:45.233757+0000', 'last_peered': '2024-09-11T05:02:45.233757+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:45.233757+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8921, 'log_dups_size': 0, 'ondisk_log_size': 8921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30568176, 'num_objects': 3732, 'num_object_clones': 0, 'num_object_copies': 14928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8918, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1434}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:50.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8976", 'reported_seq': 9068, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.752065+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:45.752065+0000', 'last_peered': '2024-09-11T05:02:45.752065+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:45.752065+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8976, 'log_dups_size': 0, 'ondisk_log_size': 8976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30693968, 'num_objects': 3747, 'num_object_clones': 0, 'num_object_copies': 14988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8975, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:50.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:50.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8901", 'reported_seq': 9003, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.202484+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:45.202484+0000', 'last_peered': '2024-09-11T05:02:45.202484+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:45.202484+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8901, 'log_dups_size': 0, 'ondisk_log_size': 8901, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30233760, 'num_objects': 3691, 'num_object_clones': 0, 'num_object_copies': 14764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8899, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1427}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:50.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:50.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8810", 'reported_seq': 8924, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:45.844624+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:45.844624+0000', 'last_peered': '2024-09-11T05:02:45.844624+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:45.844624+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8810, 'log_dups_size': 0, 'ondisk_log_size': 8810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29480096, 'num_objects': 3599, 'num_object_clones': 0, 'num_object_copies': 14396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8808, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:50.978 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:50.978 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'9057", 'reported_seq': 9172, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:48.653427+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:48.653427+0000', 'last_peered': '2024-09-11T05:02:48.653427+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:48.653427+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9057, 'log_dups_size': 0, 'ondisk_log_size': 9057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30471328, 'num_objects': 3720, 'num_object_clones': 0, 'num_object_copies': 14880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9055, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1432}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:50.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:51.772 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:51.772 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32116/235932 objects degraded (13.612%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":498,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58983,"data_bytes":483141256,"bytes_used":2847428608,"bytes_avail":383699628032,"bytes_total":386547056640,"degraded_objects":32116,"degraded_total":235932,"degraded_ratio":0.13612396792296086,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":209},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:51.796 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:51.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:52.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:52.466 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:52.498 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-11T05:02:51.792202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476923528,"num_objects":58224,"num_object_clones":0,"num_object_copies":232896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136969,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2557,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143185,"ondisk_log_size":143185,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797696,"kb_used_data":934768,"kb_used_omap":14,"kb_used_meta":1862833,"kb_avail":374689664,"statfs":{"total":386547056640,"available":383682215936,"internally_reserved":0,"allocated":957202432,"data_stored":956482556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1907541558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,2,14,16,10,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23912448,"num_objects":-2919,"num_object_clones":0,"num_object_copies":-11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006559"},"pg_stats":[{"pgid":"1.f","version":"33'8915","reported_seq":9005,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.008637+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:50.008637+0000","last_peered":"2024-09-11T05:02:50.008637+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:50.008637+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29483008,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8915,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8986","reported_seq":9094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.842346+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:48.842346+0000","last_peered":"2024-09-11T05:02:48.842346+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:48.842346+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30733472,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8984,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8631","reported_seq":8731,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:51.022902+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:51.022902+0000","last_peered":"2024-09-11T05:02:51.022902+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:51.022902+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8631,"log_dups_size":0,"ondisk_log_size":8631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28908136,"num_objects":3530,"num_object_clones":0,"num_object_copies":14120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8630,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8829","reported_seq":8932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.578365+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:48.578365+0000","last_peered":"2024-09-11T05:02:48.578365+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:48.578365+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381792,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8885","reported_seq":9003,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.226031+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:50.226031+0000","last_peered":"2024-09-11T05:02:50.226031+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:50.226031+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8885,"log_dups_size":0,"ondisk_log_size":8885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29865120,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8883,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'9008","reported_seq":9108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.274177+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:50.274177+0000","last_peered":"2024-09-11T05:02:50.274177+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:50.274177+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9008,"log_dups_size":0,"ondisk_log_size":9008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29753344,"num_objects":3632,"num_object_clones":0,"num_object_copies":14528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9028","reported_seq":9117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.771909+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:48.771909+0000","last_peered":"2024-09-11T05:02:48.771909+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:48.771909+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9028,"log_dups_size":0,"ondisk_log_size":9028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30323872,"num_objects":3702,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9026,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9128","reported_seq":13726,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.874917+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:48.874917+0000","last_peered":"2024-09-11T05:02:48.874917+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:48.874917+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282912,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9139","reported_seq":9257,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.239547+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:50.239547+0000","last_peered":"2024-09-11T05:02:50.239547+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:50.239547+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29886784,"num_objects":3649,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9136,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8923","reported_seq":9028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.856255+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:50.856255+0000","last_peered":"2024-09-11T05:02:50.856255+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:50.856255+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29529248,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8921,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1299}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8977","reported_seq":9089,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:49.976385+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:49.976385+0000","last_peered":"2024-09-11T05:02:49.976385+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:49.976385+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8977,"log_dups_size":0,"ondisk_log_size":8977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30109424,"num_objects":3676,"num_object_clones":0,"num_object_copies":14704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8773","reported_seq":13369,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:49.892310+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:49.892310+0000","last_peered":"2024-09-11T05:02:49.892310+0000","last_clean":"2024-09-11T05:02:49.892310+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:49.892310+0000","last_undegraded":"2024-09-11T05:02:49.892310+0000","last_fullsized":"2024-09-11T05:02:49.892310+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8773,"log_dups_size":0,"ondisk_log_size":8773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29581312,"num_objects":3611,"num_object_clones":0,"num_object_copies":14444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2583,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'9045","reported_seq":9137,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.817450+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:50.817450+0000","last_peered":"2024-09-11T05:02:50.817450+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:50.817450+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30128720,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8974","reported_seq":9076,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.116937+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:50.116937+0000","last_peered":"2024-09-11T05:02:50.116937+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:50.116937+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8974,"log_dups_size":0,"ondisk_log_size":8974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29635744,"num_objects":3618,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8972,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1354}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8887","reported_seq":9001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.982368+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:50.982368+0000","last_peered":"2024-09-11T05:02:50.982368+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:50.982368+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849312,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9057","reported_seq":9172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:48.653427+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:48.653427+0000","last_peered":"2024-09-11T05:02:48.653427+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:48.653427+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471328,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9055,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":476923528,"num_objects":58224,"num_object_clones":0,"num_object_copies":232896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136969,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2557,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":954441728,"data_stored":954441728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143185,"ondisk_log_size":143185,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738524,"kb_used_data":233416,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93633316,"statfs":{"total":96636764160,"available":95880515584,"internally_reserved":0,"allocated":239017984,"data_stored":238838015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733368,"kb_used_data":232936,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93638472,"statfs":{"total":96636764160,"available":95885795328,"internally_reserved":0,"allocated":238526464,"data_stored":238346495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,4,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594600,"kb_used_data":233932,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93777240,"statfs":{"total":96636764160,"available":96027893760,"internally_reserved":0,"allocated":239546368,"data_stored":239366399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":369290962},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,3,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731204,"kb_used_data":234484,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93640636,"statfs":{"total":96636764160,"available":95888011264,"internally_reserved":0,"allocated":240111616,"data_stored":239931647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,8,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239431680,"data_stored":239431680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239222784,"data_stored":239222784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237608960,"data_stored":237608960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238178304,"data_stored":238178304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:55.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:56.176 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:56.206 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-11T05:02:55.793577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":470517384,"num_objects":57442,"num_object_clones":0,"num_object_copies":229768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57442,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137751,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143967,"ondisk_log_size":143967,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2798260,"kb_used_data":922868,"kb_used_omap":14,"kb_used_meta":1875313,"kb_avail":374689100,"statfs":{"total":386547056640,"available":383681638400,"internally_reserved":0,"allocated":945016832,"data_stored":944296956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1920321078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,8,5,7,12,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21454848,"num_objects":-2619,"num_object_clones":0,"num_object_copies":-10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004899"},"pg_stats":[{"pgid":"1.f","version":"33'8975","reported_seq":9065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.325268+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:55.325268+0000","last_peered":"2024-09-11T05:02:55.325268+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:55.325268+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28991488,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9056","reported_seq":9166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.712603+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:53.712603+0000","last_peered":"2024-09-11T05:02:53.712603+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:53.712603+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160032,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8631","reported_seq":8731,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:51.022902+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:51.022902+0000","last_peered":"2024-09-11T05:02:51.022902+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:51.022902+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8631,"log_dups_size":0,"ondisk_log_size":8631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28908136,"num_objects":3530,"num_object_clones":0,"num_object_copies":14120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8630,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8890","reported_seq":8993,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.712243+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:53.712243+0000","last_peered":"2024-09-11T05:02:53.712243+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:53.712243+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28882080,"num_objects":3526,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8961","reported_seq":9079,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.355225+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:55.355225+0000","last_peered":"2024-09-11T05:02:55.355225+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:55.355225+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8959,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'9067","reported_seq":9168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.322096+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:55.322096+0000","last_peered":"2024-09-11T05:02:55.322096+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:55.322096+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9067,"log_dups_size":0,"ondisk_log_size":9067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29270016,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9100","reported_seq":9188,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.936571+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:53.936571+0000","last_peered":"2024-09-11T05:02:53.936571+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:53.936571+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9100,"log_dups_size":0,"ondisk_log_size":9100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734048,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9184","reported_seq":13899,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.935503+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:53.935503+0000","last_peered":"2024-09-11T05:02:53.935503+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:53.935503+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9184,"log_dups_size":0,"ondisk_log_size":9184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29824160,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2325,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9202","reported_seq":9320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.492106+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:55.492106+0000","last_peered":"2024-09-11T05:02:55.492106+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:55.492106+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9202,"log_dups_size":0,"ondisk_log_size":9202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29370688,"num_objects":3586,"num_object_clones":0,"num_object_copies":14344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8923","reported_seq":9028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.856255+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:50.856255+0000","last_peered":"2024-09-11T05:02:50.856255+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:50.856255+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29529248,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8921,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1299}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'9047","reported_seq":9159,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.470799+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:55.470799+0000","last_peered":"2024-09-11T05:02:55.470799+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:55.470799+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9047,"log_dups_size":0,"ondisk_log_size":9047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29535984,"num_objects":3606,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8845","reported_seq":13441,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:54.977845+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:54.977845+0000","last_peered":"2024-09-11T05:02:54.977845+0000","last_clean":"2024-09-11T05:02:54.977845+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:54.977845+0000","last_undegraded":"2024-09-11T05:02:54.977845+0000","last_fullsized":"2024-09-11T05:02:54.977845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28991488,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'9045","reported_seq":9137,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.817450+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:50.817450+0000","last_peered":"2024-09-11T05:02:50.817450+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:50.817450+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30128720,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9031","reported_seq":9133,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.410362+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:55.410362+0000","last_peered":"2024-09-11T05:02:55.410362+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:55.410362+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29168800,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1297}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8887","reported_seq":9001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.982368+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:50.982368+0000","last_peered":"2024-09-11T05:02:50.982368+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:50.982368+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849312,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9123","reported_seq":9238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.862265+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:53.862265+0000","last_peered":"2024-09-11T05:02:53.862265+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:53.862265+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9123,"log_dups_size":0,"ondisk_log_size":9123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29930656,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9121,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":470517384,"num_objects":57442,"num_object_clones":0,"num_object_copies":229768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57442,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137751,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":941174784,"data_stored":941174784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143967,"ondisk_log_size":143967,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738484,"kb_used_data":229220,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93633356,"statfs":{"total":96636764160,"available":95880556544,"internally_reserved":0,"allocated":234721280,"data_stored":234541311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,0,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733368,"kb_used_data":232936,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93638472,"statfs":{"total":96636764160,"available":95885795328,"internally_reserved":0,"allocated":238526464,"data_stored":238346495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,4,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595300,"kb_used_data":230484,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93776540,"statfs":{"total":96636764160,"available":96027176960,"internally_reserved":0,"allocated":236015616,"data_stored":235835647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":373550802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731108,"kb_used_data":230228,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93640732,"statfs":{"total":96636764160,"available":95888109568,"internally_reserved":0,"allocated":235753472,"data_stored":235573503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,2,3,0,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235089920,"data_stored":235089920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234856448,"data_stored":234856448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237608960,"data_stored":237608960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233619456,"data_stored":233619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:56.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:56.865 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:56.866 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:56.897 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-11T05:02:55.793577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":470517384,"num_objects":57442,"num_object_clones":0,"num_object_copies":229768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57442,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137751,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143967,"ondisk_log_size":143967,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2798260,"kb_used_data":922868,"kb_used_omap":14,"kb_used_meta":1875313,"kb_avail":374689100,"statfs":{"total":386547056640,"available":383681638400,"internally_reserved":0,"allocated":945016832,"data_stored":944296956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1920321078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,8,5,7,12,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21454848,"num_objects":-2619,"num_object_clones":0,"num_object_copies":-10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004899"},"pg_stats":[{"pgid":"1.f","version":"33'8975","reported_seq":9065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.325268+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:55.325268+0000","last_peered":"2024-09-11T05:02:55.325268+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:55.325268+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28991488,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9056","reported_seq":9166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.712603+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:53.712603+0000","last_peered":"2024-09-11T05:02:53.712603+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:53.712603+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160032,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8631","reported_seq":8731,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:51.022902+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:51.022902+0000","last_peered":"2024-09-11T05:02:51.022902+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:51.022902+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8631,"log_dups_size":0,"ondisk_log_size":8631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28908136,"num_objects":3530,"num_object_clones":0,"num_object_copies":14120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8630,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8890","reported_seq":8993,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.712243+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:53.712243+0000","last_peered":"2024-09-11T05:02:53.712243+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:53.712243+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28882080,"num_objects":3526,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8961","reported_seq":9079,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.355225+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:55.355225+0000","last_peered":"2024-09-11T05:02:55.355225+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:55.355225+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8959,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'9067","reported_seq":9168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.322096+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:55.322096+0000","last_peered":"2024-09-11T05:02:55.322096+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:55.322096+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9067,"log_dups_size":0,"ondisk_log_size":9067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29270016,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9100","reported_seq":9188,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.936571+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:53.936571+0000","last_peered":"2024-09-11T05:02:53.936571+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:53.936571+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9100,"log_dups_size":0,"ondisk_log_size":9100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734048,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9184","reported_seq":13899,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.935503+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:53.935503+0000","last_peered":"2024-09-11T05:02:53.935503+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:53.935503+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9184,"log_dups_size":0,"ondisk_log_size":9184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29824160,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2325,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9202","reported_seq":9320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.492106+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:55.492106+0000","last_peered":"2024-09-11T05:02:55.492106+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:55.492106+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9202,"log_dups_size":0,"ondisk_log_size":9202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29370688,"num_objects":3586,"num_object_clones":0,"num_object_copies":14344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8923","reported_seq":9028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.856255+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:50.856255+0000","last_peered":"2024-09-11T05:02:50.856255+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:50.856255+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29529248,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8921,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1299}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'9047","reported_seq":9159,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.470799+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:55.470799+0000","last_peered":"2024-09-11T05:02:55.470799+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:55.470799+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9047,"log_dups_size":0,"ondisk_log_size":9047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29535984,"num_objects":3606,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8845","reported_seq":13441,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:54.977845+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:54.977845+0000","last_peered":"2024-09-11T05:02:54.977845+0000","last_clean":"2024-09-11T05:02:54.977845+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:54.977845+0000","last_undegraded":"2024-09-11T05:02:54.977845+0000","last_fullsized":"2024-09-11T05:02:54.977845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28991488,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'9045","reported_seq":9137,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.817450+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:50.817450+0000","last_peered":"2024-09-11T05:02:50.817450+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:50.817450+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30128720,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9031","reported_seq":9133,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.410362+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:55.410362+0000","last_peered":"2024-09-11T05:02:55.410362+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:55.410362+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29168800,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1297}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8887","reported_seq":9001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:50.982368+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:50.982368+0000","last_peered":"2024-09-11T05:02:50.982368+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:50.982368+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849312,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9123","reported_seq":9238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.862265+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:53.862265+0000","last_peered":"2024-09-11T05:02:53.862265+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:53.862265+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9123,"log_dups_size":0,"ondisk_log_size":9123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29930656,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9121,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":470517384,"num_objects":57442,"num_object_clones":0,"num_object_copies":229768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57442,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137751,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":941174784,"data_stored":941174784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143967,"ondisk_log_size":143967,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738484,"kb_used_data":229220,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93633356,"statfs":{"total":96636764160,"available":95880556544,"internally_reserved":0,"allocated":234721280,"data_stored":234541311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,0,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733368,"kb_used_data":232936,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93638472,"statfs":{"total":96636764160,"available":95885795328,"internally_reserved":0,"allocated":238526464,"data_stored":238346495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,4,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595300,"kb_used_data":230484,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93776540,"statfs":{"total":96636764160,"available":96027176960,"internally_reserved":0,"allocated":236015616,"data_stored":235835647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":373550802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731108,"kb_used_data":230228,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93640732,"statfs":{"total":96636764160,"available":95888109568,"internally_reserved":0,"allocated":235753472,"data_stored":235573503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,2,3,0,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235089920,"data_stored":235089920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234856448,"data_stored":234856448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237608960,"data_stored":237608960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233619456,"data_stored":233619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:56.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:02:56.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8975", 'reported_seq': 9065, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:55.325268+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:02:55.325268+0000', 'last_peered': '2024-09-11T05:02:55.325268+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:02:55.325268+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8975, 'log_dups_size': 0, 'ondisk_log_size': 8975, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28991488, 'num_objects': 3539, 'num_object_clones': 0, 'num_object_copies': 14156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8975, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1248}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:56.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:02:56.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'9056", 'reported_seq': 9166, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:53.712603+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:53.712603+0000', 'last_peered': '2024-09-11T05:02:53.712603+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:53.712603+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9056, 'log_dups_size': 0, 'ondisk_log_size': 9056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30160032, 'num_objects': 3682, 'num_object_clones': 0, 'num_object_copies': 14728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9054, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:56.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:02:56.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8631", 'reported_seq': 8731, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:51.022902+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:02:51.022902+0000', 'last_peered': '2024-09-11T05:02:51.022902+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:02:51.022902+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8631, 'log_dups_size': 0, 'ondisk_log_size': 8631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28908136, 'num_objects': 3530, 'num_object_clones': 0, 'num_object_copies': 14120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3530, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8630, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1319}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:56.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:02:56.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8890", 'reported_seq': 8993, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:53.712243+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:53.712243+0000', 'last_peered': '2024-09-11T05:02:53.712243+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:53.712243+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8890, 'log_dups_size': 0, 'ondisk_log_size': 8890, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28882080, 'num_objects': 3526, 'num_object_clones': 0, 'num_object_copies': 14104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8888, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8961", 'reported_seq': 9079, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:55.355225+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:02:55.355225+0000', 'last_peered': '2024-09-11T05:02:55.355225+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:02:55.355225+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8961, 'log_dups_size': 0, 'ondisk_log_size': 8961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29242528, 'num_objects': 3570, 'num_object_clones': 0, 'num_object_copies': 14280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8959, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1349}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'9067", 'reported_seq': 9168, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:55.322096+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:02:55.322096+0000', 'last_peered': '2024-09-11T05:02:55.322096+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:02:55.322096+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9067, 'log_dups_size': 0, 'ondisk_log_size': 9067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29270016, 'num_objects': 3573, 'num_object_clones': 0, 'num_object_copies': 14292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9067, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1279}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'9100", 'reported_seq': 9188, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:53.936571+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:53.936571+0000', 'last_peered': '2024-09-11T05:02:53.936571+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:53.936571+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9100, 'log_dups_size': 0, 'ondisk_log_size': 9100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29734048, 'num_objects': 3630, 'num_object_clones': 0, 'num_object_copies': 14520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9098, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1323}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9184", 'reported_seq': 13899, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:53.935503+0000', 'last_change': '2024-09-11T04:59:38.581267+0000', 'last_active': '2024-09-11T05:02:53.935503+0000', 'last_peered': '2024-09-11T05:02:53.935503+0000', 'last_clean': '2024-09-11T04:55:08.509099+0000', 'last_became_active': '2024-09-11T04:56:20.526554+0000', 'last_became_peered': '2024-09-11T04:56:20.526554+0000', 'last_unstale': '2024-09-11T05:02:53.935503+0000', 'last_undegraded': '2024-09-11T04:56:20.285667+0000', 'last_fullsized': '2024-09-11T04:56:20.223958+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9184, 'log_dups_size': 0, 'ondisk_log_size': 9184, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29824160, 'num_objects': 3641, 'num_object_clones': 0, 'num_object_copies': 14564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 40, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9182, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2325, 'num_bytes_recovered': 15564800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:56.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'9202", 'reported_seq': 9320, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:55.492106+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:02:55.492106+0000', 'last_peered': '2024-09-11T05:02:55.492106+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:02:55.492106+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9202, 'log_dups_size': 0, 'ondisk_log_size': 9202, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29370688, 'num_objects': 3586, 'num_object_clones': 0, 'num_object_copies': 14344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9199, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1220}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8923", 'reported_seq': 9028, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:50.856255+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:02:50.856255+0000', 'last_peered': '2024-09-11T05:02:50.856255+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:02:50.856255+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8923, 'log_dups_size': 0, 'ondisk_log_size': 8923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29529248, 'num_objects': 3605, 'num_object_clones': 0, 'num_object_copies': 14420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8921, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1299}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'9047", 'reported_seq': 9159, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:55.470799+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:02:55.470799+0000', 'last_peered': '2024-09-11T05:02:55.470799+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:02:55.470799+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9047, 'log_dups_size': 0, 'ondisk_log_size': 9047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29535984, 'num_objects': 3606, 'num_object_clones': 0, 'num_object_copies': 14424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9044, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1308}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'9045", 'reported_seq': 9137, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:50.817450+0000', 'last_change': '2024-09-11T04:56:20.565489+0000', 'last_active': '2024-09-11T05:02:50.817450+0000', 'last_peered': '2024-09-11T05:02:50.817450+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:02:50.817450+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9045, 'log_dups_size': 0, 'ondisk_log_size': 9045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30128720, 'num_objects': 3678, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9044, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'9031", 'reported_seq': 9133, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:55.410362+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:02:55.410362+0000', 'last_peered': '2024-09-11T05:02:55.410362+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:02:55.410362+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9031, 'log_dups_size': 0, 'ondisk_log_size': 9031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29168800, 'num_objects': 3561, 'num_object_clones': 0, 'num_object_copies': 14244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9029, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1297}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:02:56.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:02:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8887", 'reported_seq': 9001, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:50.982368+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:02:50.982368+0000', 'last_peered': '2024-09-11T05:02:50.982368+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:02:50.982368+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8887, 'log_dups_size': 0, 'ondisk_log_size': 8887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28849312, 'num_objects': 3522, 'num_object_clones': 0, 'num_object_copies': 14088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8885, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:02:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:02:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'9123", 'reported_seq': 9238, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:53.862265+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:53.862265+0000', 'last_peered': '2024-09-11T05:02:53.862265+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:53.862265+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9123, 'log_dups_size': 0, 'ondisk_log_size': 9123, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29930656, 'num_objects': 3654, 'num_object_clones': 0, 'num_object_copies': 14616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3654, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9121, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:02:56.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:02:57.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:57.747 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32058/229768 objects degraded (13.952%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":504,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57442,"data_bytes":470517384,"bytes_used":2865418240,"bytes_avail":383681638400,"bytes_total":386547056640,"degraded_objects":32058,"degraded_total":229768,"degraded_ratio":0.13952334528742036,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":218},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:02:57.776 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:02:57.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:02:58.382 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:02:58.383 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:02:58.420 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-11T05:02:57.859860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468100744,"num_objects":57147,"num_object_clones":0,"num_object_copies":228588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138046,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144262,"ondisk_log_size":144262,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2798160,"kb_used_data":918608,"kb_used_omap":14,"kb_used_meta":1879473,"kb_avail":374689200,"statfs":{"total":386547056640,"available":383681740800,"internally_reserved":0,"allocated":940654592,"data_stored":939934716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1924580918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,1,8,13,5,8,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17866752,"num_objects":-2181,"num_object_clones":0,"num_object_copies":-8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2181,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.069394"},"pg_stats":[{"pgid":"1.f","version":"33'8975","reported_seq":9065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.325268+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:02:55.325268+0000","last_peered":"2024-09-11T05:02:55.325268+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:02:55.325268+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28991488,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9056","reported_seq":9166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.712603+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:53.712603+0000","last_peered":"2024-09-11T05:02:53.712603+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:53.712603+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160032,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8705","reported_seq":8805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.967064+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:02:55.967064+0000","last_peered":"2024-09-11T05:02:55.967064+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:02:55.967064+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8705,"log_dups_size":0,"ondisk_log_size":8705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28301928,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8704,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1245}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8890","reported_seq":8993,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.712243+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:53.712243+0000","last_peered":"2024-09-11T05:02:53.712243+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:53.712243+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28882080,"num_objects":3526,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8961","reported_seq":9079,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.355225+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:02:55.355225+0000","last_peered":"2024-09-11T05:02:55.355225+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:02:55.355225+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8959,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'9067","reported_seq":9168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.322096+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:02:55.322096+0000","last_peered":"2024-09-11T05:02:55.322096+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:02:55.322096+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9067,"log_dups_size":0,"ondisk_log_size":9067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29270016,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9100","reported_seq":9188,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.936571+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:53.936571+0000","last_peered":"2024-09-11T05:02:53.936571+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:53.936571+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9100,"log_dups_size":0,"ondisk_log_size":9100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734048,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9184","reported_seq":13899,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.935503+0000","last_change":"2024-09-11T04:59:38.581267+0000","last_active":"2024-09-11T05:02:53.935503+0000","last_peered":"2024-09-11T05:02:53.935503+0000","last_clean":"2024-09-11T04:55:08.509099+0000","last_became_active":"2024-09-11T04:56:20.526554+0000","last_became_peered":"2024-09-11T04:56:20.526554+0000","last_unstale":"2024-09-11T05:02:53.935503+0000","last_undegraded":"2024-09-11T04:56:20.285667+0000","last_fullsized":"2024-09-11T04:56:20.223958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9184,"log_dups_size":0,"ondisk_log_size":9184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29824160,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2325,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9202","reported_seq":9320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.492106+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:02:55.492106+0000","last_peered":"2024-09-11T05:02:55.492106+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:02:55.492106+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9202,"log_dups_size":0,"ondisk_log_size":9202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29370688,"num_objects":3586,"num_object_clones":0,"num_object_copies":14344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8991","reported_seq":9097,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.997862+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:02:55.997862+0000","last_peered":"2024-09-11T05:02:55.997862+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:02:55.997862+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28972192,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8989,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'9047","reported_seq":9159,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.470799+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:02:55.470799+0000","last_peered":"2024-09-11T05:02:55.470799+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:02:55.470799+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9047,"log_dups_size":0,"ondisk_log_size":9047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29535984,"num_objects":3606,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8845","reported_seq":13441,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-11T05:02:54.977845+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:02:54.977845+0000","last_peered":"2024-09-11T05:02:54.977845+0000","last_clean":"2024-09-11T05:02:54.977845+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:02:54.977845+0000","last_undegraded":"2024-09-11T05:02:54.977845+0000","last_fullsized":"2024-09-11T05:02:54.977845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28991488,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'9126","reported_seq":9218,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.930971+0000","last_change":"2024-09-11T04:56:20.565489+0000","last_active":"2024-09-11T05:02:55.930971+0000","last_peered":"2024-09-11T05:02:55.930971+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:02:55.930971+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29465168,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1271}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9031","reported_seq":9133,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.410362+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:02:55.410362+0000","last_peered":"2024-09-11T05:02:55.410362+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:02:55.410362+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29168800,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1297}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8959","reported_seq":9073,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:55.977846+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:02:55.977846+0000","last_peered":"2024-09-11T05:02:55.977846+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:02:55.977846+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8959,"log_dups_size":0,"ondisk_log_size":8959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28259488,"num_objects":3450,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8957,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9123","reported_seq":9238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:53.862265+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:53.862265+0000","last_peered":"2024-09-11T05:02:53.862265+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:53.862265+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9123,"log_dups_size":0,"ondisk_log_size":9123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29930656,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9121,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":468100744,"num_objects":57147,"num_object_clones":0,"num_object_copies":228588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138046,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2615,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936767488,"data_stored":936767488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144262,"ondisk_log_size":144262,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738484,"kb_used_data":229220,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93633356,"statfs":{"total":96636764160,"available":95880556544,"internally_reserved":0,"allocated":234721280,"data_stored":234541311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,0,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733268,"kb_used_data":228676,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93638572,"statfs":{"total":96636764160,"available":95885897728,"internally_reserved":0,"allocated":234164224,"data_stored":233984255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,10,2,0,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595300,"kb_used_data":230484,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93776540,"statfs":{"total":96636764160,"available":96027176960,"internally_reserved":0,"allocated":236015616,"data_stored":235835647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":373550802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731108,"kb_used_data":230228,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93640732,"statfs":{"total":96636764160,"available":95888109568,"internally_reserved":0,"allocated":235753472,"data_stored":235573503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,2,3,0,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235089920,"data_stored":235089920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234856448,"data_stored":234856448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233201664,"data_stored":233201664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233619456,"data_stored":233619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:02:59.184 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:02:59.183+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9243 (0'0,35'9243] local-lis/les=34/35 n=3582 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9235 crt=35'9235 lcod 35'9234 mlcod 35'9234 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:02:59.486 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:02:59.475+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9245 (0'0,35'9245] local-lis/les=34/35 n=3580 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9245 lcod 35'9244 mlcod 35'9244 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:02:59.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:02:59.712+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9251 (0'0,35'9251] local-lis/les=34/35 n=3574 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9248 crt=35'9247 lcod 35'9247 mlcod 35'9247 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:02:59.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:02:59.784+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9251 (0'0,35'9251] local-lis/les=34/35 n=3574 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9251 lcod 35'9250 mlcod 35'9250 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:02:59.965 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:02:59.961+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9253 (0'0,35'9253] local-lis/les=34/35 n=3572 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9252 crt=35'9251 lcod 35'9251 mlcod 35'9251 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:00.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:00.141+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9255 (0'0,35'9255] local-lis/les=34/35 n=3570 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9253 crt=35'9253 lcod 35'9252 mlcod 35'9252 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:00.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:00.377+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9262 (0'0,35'9262] local-lis/les=34/35 n=3563 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9260 crt=35'9259 lcod 35'9259 mlcod 35'9259 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:00.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:00.563+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9263 (0'0,35'9263] local-lis/les=34/35 n=3562 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9263 lcod 35'9262 mlcod 35'9262 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:00.732 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:00.730+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9266 (0'0,35'9266] local-lis/les=34/35 n=3559 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9265 crt=35'9265 lcod 35'9264 mlcod 35'9264 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:00.824 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:00.823+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9267 (0'0,35'9267] local-lis/les=34/35 n=3558 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9266 crt=35'9266 lcod 35'9265 mlcod 35'9265 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:00.964 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:00.959+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9268 (0'0,35'9268] local-lis/les=34/35 n=3557 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9268 lcod 35'9267 mlcod 35'9267 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:01.231 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:01.226+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9270 (0'0,35'9270] local-lis/les=34/35 n=3555 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9269 crt=35'9269 lcod 35'9268 mlcod 35'9268 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:01.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:01.464 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:01.447+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9272 (0'0,35'9272] local-lis/les=34/35 n=3553 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9270 crt=35'9270 lcod 35'9269 mlcod 35'9269 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:01.590 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:01.590+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9272 (0'0,35'9272] local-lis/les=34/35 n=3553 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9272 lcod 35'9271 mlcod 35'9271 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:01.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:01.644+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9273 (0'0,35'9273] local-lis/les=34/35 n=3552 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9272 crt=35'9272 lcod 35'9271 mlcod 35'9271 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:01.857 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:01.856+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9275 (0'0,35'9275] local-lis/les=34/35 n=3550 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9275 lcod 35'9274 mlcod 35'9274 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:02.083 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:02.115 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-11T05:03:01.795835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459695752,"num_objects":56121,"num_object_clones":0,"num_object_copies":224484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139072,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2700,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145288,"ondisk_log_size":145288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819040,"kb_used_data":901912,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374668320,"statfs":{"total":386547056640,"available":383660359680,"internally_reserved":0,"allocated":923557888,"data_stored":922805707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1962985014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,11,10,9,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17227776,"num_objects":-2103,"num_object_clones":0,"num_object_copies":-8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002944"},"pg_stats":[{"pgid":"1.f","version":"35'9036","reported_seq":9130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.091251+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:00.091251+0000","last_peered":"2024-09-11T05:03:00.091251+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:00.091251+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28491776,"num_objects":3478,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'9124","reported_seq":9238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.883247+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:58.883247+0000","last_peered":"2024-09-11T05:02:58.883247+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:58.883247+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29602976,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8778","reported_seq":8880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.141294+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:01.141294+0000","last_peered":"2024-09-11T05:03:01.141294+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:01.141294+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8778,"log_dups_size":0,"ondisk_log_size":8778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27703912,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8777,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1172}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'8955","reported_seq":9062,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.881980+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:58.881980+0000","last_peered":"2024-09-11T05:02:58.881980+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:58.881980+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28349600,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9012","reported_seq":9134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.082894+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:00.082894+0000","last_peered":"2024-09-11T05:03:00.082894+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:00.082894+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9012,"log_dups_size":0,"ondisk_log_size":9012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28824736,"num_objects":3519,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9010,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9129","reported_seq":9234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.549345+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:00.549345+0000","last_peered":"2024-09-11T05:03:00.549345+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:00.549345+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9129,"log_dups_size":0,"ondisk_log_size":9129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28762112,"num_objects":3511,"num_object_clones":0,"num_object_copies":14044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9129,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1217}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9175","reported_seq":9268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.889548+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:58.889548+0000","last_peered":"2024-09-11T05:02:58.889548+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:58.889548+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119648,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1248}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9255","reported_seq":14072,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T05:03:00.015578+0000","last_change":"2024-09-11T05:02:59.080607+0000","last_active":"2024-09-11T05:03:00.015578+0000","last_peered":"2024-09-11T05:03:00.015578+0000","last_clean":"2024-09-11T05:03:00.015578+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:00.015578+0000","last_undegraded":"2024-09-11T05:03:00.015578+0000","last_fullsized":"2024-09-11T05:03:00.015578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":436,"log_size":9255,"log_dups_size":0,"ondisk_log_size":9255,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9253,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2365,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9261","reported_seq":9383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.467696+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:00.467696+0000","last_peered":"2024-09-11T05:03:00.467696+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:00.467696+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9261,"log_dups_size":0,"ondisk_log_size":9261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28887360,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9258,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9048","reported_seq":9159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.955513+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:00.955513+0000","last_peered":"2024-09-11T05:03:00.955513+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:00.955513+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9048,"log_dups_size":0,"ondisk_log_size":9048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1174}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9119","reported_seq":9235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.359339+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:00.359339+0000","last_peered":"2024-09-11T05:03:00.359339+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:00.359339+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9119,"log_dups_size":0,"ondisk_log_size":9119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28946160,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8908","reported_seq":13508,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:00.077126+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:00.077126+0000","last_peered":"2024-09-11T05:03:00.077126+0000","last_clean":"2024-09-11T05:03:00.077126+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:00.077126+0000","last_undegraded":"2024-09-11T05:03:00.077126+0000","last_fullsized":"2024-09-11T05:03:00.077126+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8908,"log_dups_size":0,"ondisk_log_size":8908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28475392,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9189","reported_seq":9376,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.176915+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:01.176915+0000","last_peered":"2024-09-11T05:03:01.176915+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:01.176915+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9189,"log_dups_size":0,"ondisk_log_size":9189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28949072,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9188,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9099","reported_seq":9205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.428223+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:00.428223+0000","last_peered":"2024-09-11T05:03:00.428223+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:00.428223+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28611744,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1229}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9020","reported_seq":9137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.074928+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:01.074928+0000","last_peered":"2024-09-11T05:03:01.074928+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:01.074928+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9020,"log_dups_size":0,"ondisk_log_size":9020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27759776,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'9180","reported_seq":9299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.888924+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:58.888924+0000","last_peered":"2024-09-11T05:02:58.888924+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:58.888924+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9180,"log_dups_size":0,"ondisk_log_size":9180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9178,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459695752,"num_objects":56121,"num_object_clones":0,"num_object_copies":224484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139072,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2700,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":920215552,"data_stored":920215552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145288,"ondisk_log_size":145288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":225256,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":230662144,"data_stored":230472942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733572,"kb_used_data":224804,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93638268,"statfs":{"total":96636764160,"available":95885586432,"internally_reserved":0,"allocated":230199296,"data_stored":230010094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632080,"kb_used_data":225592,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93739760,"statfs":{"total":96636764160,"available":95989514240,"internally_reserved":0,"allocated":231006208,"data_stored":230817006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416214738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731312,"kb_used_data":226260,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93640528,"statfs":{"total":96636764160,"available":95887900672,"internally_reserved":0,"allocated":231690240,"data_stored":231505665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,1,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230957056,"data_stored":230957056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230633472,"data_stored":230633472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229036032,"data_stored":229036032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229588992,"data_stored":229588992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:02.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:02.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:02.131+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9279 (0'0,35'9279] local-lis/les=34/35 n=3546 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9276 crt=35'9276 lcod 35'9275 mlcod 35'9275 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:02.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:02.227+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9281 (0'0,35'9281] local-lis/les=34/35 n=3544 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9280 crt=35'9280 lcod 35'9279 mlcod 35'9279 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:02.388 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:02.382+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9281 (0'0,35'9281] local-lis/les=34/35 n=3544 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9281 lcod 35'9280 mlcod 35'9280 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:02.466 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:02.465+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9282 (0'0,35'9282] local-lis/les=34/35 n=3543 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9281 crt=35'9281 lcod 35'9280 mlcod 35'9280 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:02.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:02.709+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9283 (0'0,35'9283] local-lis/les=34/35 n=3542 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9282 crt=35'9282 lcod 35'9281 mlcod 35'9281 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:02.815 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:02.836 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-11T05:03:01.795835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459695752,"num_objects":56121,"num_object_clones":0,"num_object_copies":224484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139072,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2700,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145288,"ondisk_log_size":145288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819040,"kb_used_data":901912,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374668320,"statfs":{"total":386547056640,"available":383660359680,"internally_reserved":0,"allocated":923557888,"data_stored":922805707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1962985014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,11,10,9,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17227776,"num_objects":-2103,"num_object_clones":0,"num_object_copies":-8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002944"},"pg_stats":[{"pgid":"1.f","version":"35'9036","reported_seq":9130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.091251+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:00.091251+0000","last_peered":"2024-09-11T05:03:00.091251+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:00.091251+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28491776,"num_objects":3478,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'9124","reported_seq":9238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.883247+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:58.883247+0000","last_peered":"2024-09-11T05:02:58.883247+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:58.883247+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29602976,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8778","reported_seq":8880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.141294+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:01.141294+0000","last_peered":"2024-09-11T05:03:01.141294+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:01.141294+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8778,"log_dups_size":0,"ondisk_log_size":8778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27703912,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8777,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1172}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'8955","reported_seq":9062,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.881980+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:58.881980+0000","last_peered":"2024-09-11T05:02:58.881980+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:58.881980+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28349600,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9012","reported_seq":9134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.082894+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:00.082894+0000","last_peered":"2024-09-11T05:03:00.082894+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:00.082894+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9012,"log_dups_size":0,"ondisk_log_size":9012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28824736,"num_objects":3519,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9010,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9129","reported_seq":9234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.549345+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:00.549345+0000","last_peered":"2024-09-11T05:03:00.549345+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:00.549345+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9129,"log_dups_size":0,"ondisk_log_size":9129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28762112,"num_objects":3511,"num_object_clones":0,"num_object_copies":14044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9129,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1217}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9175","reported_seq":9268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.889548+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:58.889548+0000","last_peered":"2024-09-11T05:02:58.889548+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:58.889548+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119648,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1248}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9255","reported_seq":14072,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T05:03:00.015578+0000","last_change":"2024-09-11T05:02:59.080607+0000","last_active":"2024-09-11T05:03:00.015578+0000","last_peered":"2024-09-11T05:03:00.015578+0000","last_clean":"2024-09-11T05:03:00.015578+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:00.015578+0000","last_undegraded":"2024-09-11T05:03:00.015578+0000","last_fullsized":"2024-09-11T05:03:00.015578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":436,"log_size":9255,"log_dups_size":0,"ondisk_log_size":9255,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9253,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2365,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9261","reported_seq":9383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.467696+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:00.467696+0000","last_peered":"2024-09-11T05:03:00.467696+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:00.467696+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9261,"log_dups_size":0,"ondisk_log_size":9261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28887360,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9258,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9048","reported_seq":9159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.955513+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:00.955513+0000","last_peered":"2024-09-11T05:03:00.955513+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:00.955513+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9048,"log_dups_size":0,"ondisk_log_size":9048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1174}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9119","reported_seq":9235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.359339+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:00.359339+0000","last_peered":"2024-09-11T05:03:00.359339+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:00.359339+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9119,"log_dups_size":0,"ondisk_log_size":9119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28946160,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8908","reported_seq":13508,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:00.077126+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:00.077126+0000","last_peered":"2024-09-11T05:03:00.077126+0000","last_clean":"2024-09-11T05:03:00.077126+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:00.077126+0000","last_undegraded":"2024-09-11T05:03:00.077126+0000","last_fullsized":"2024-09-11T05:03:00.077126+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8908,"log_dups_size":0,"ondisk_log_size":8908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28475392,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9189","reported_seq":9376,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.176915+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:01.176915+0000","last_peered":"2024-09-11T05:03:01.176915+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:01.176915+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9189,"log_dups_size":0,"ondisk_log_size":9189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28949072,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9188,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9099","reported_seq":9205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.428223+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:00.428223+0000","last_peered":"2024-09-11T05:03:00.428223+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:00.428223+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28611744,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1229}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9020","reported_seq":9137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.074928+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:01.074928+0000","last_peered":"2024-09-11T05:03:01.074928+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:01.074928+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9020,"log_dups_size":0,"ondisk_log_size":9020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27759776,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'9180","reported_seq":9299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.888924+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:58.888924+0000","last_peered":"2024-09-11T05:02:58.888924+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:58.888924+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9180,"log_dups_size":0,"ondisk_log_size":9180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9178,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459695752,"num_objects":56121,"num_object_clones":0,"num_object_copies":224484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139072,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2700,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":920215552,"data_stored":920215552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145288,"ondisk_log_size":145288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":225256,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":230662144,"data_stored":230472942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733572,"kb_used_data":224804,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93638268,"statfs":{"total":96636764160,"available":95885586432,"internally_reserved":0,"allocated":230199296,"data_stored":230010094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632080,"kb_used_data":225592,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93739760,"statfs":{"total":96636764160,"available":95989514240,"internally_reserved":0,"allocated":231006208,"data_stored":230817006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416214738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731312,"kb_used_data":226260,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93640528,"statfs":{"total":96636764160,"available":95887900672,"internally_reserved":0,"allocated":231690240,"data_stored":231505665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,1,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230957056,"data_stored":230957056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230633472,"data_stored":230633472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229036032,"data_stored":229036032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229588992,"data_stored":229588992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:02.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9036", 'reported_seq': 9130, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:00.091251+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:00.091251+0000', 'last_peered': '2024-09-11T05:03:00.091251+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:00.091251+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9036, 'log_dups_size': 0, 'ondisk_log_size': 9036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28491776, 'num_objects': 3478, 'num_object_clones': 0, 'num_object_copies': 13912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9036, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1187}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'9124", 'reported_seq': 9238, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:58.883247+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:02:58.883247+0000', 'last_peered': '2024-09-11T05:02:58.883247+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:02:58.883247+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9124, 'log_dups_size': 0, 'ondisk_log_size': 9124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29602976, 'num_objects': 3614, 'num_object_clones': 0, 'num_object_copies': 14456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9122, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1356}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8778", 'reported_seq': 8880, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:01.141294+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:01.141294+0000', 'last_peered': '2024-09-11T05:03:01.141294+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:01.141294+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8778, 'log_dups_size': 0, 'ondisk_log_size': 8778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27703912, 'num_objects': 3383, 'num_object_clones': 0, 'num_object_copies': 13532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3383, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8777, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1172}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'8955", 'reported_seq': 9062, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:58.881980+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:02:58.881980+0000', 'last_peered': '2024-09-11T05:02:58.881980+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:02:58.881980+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8955, 'log_dups_size': 0, 'ondisk_log_size': 8955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28349600, 'num_objects': 3461, 'num_object_clones': 0, 'num_object_copies': 13844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8953, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1200}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:02.838 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9012", 'reported_seq': 9134, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:00.082894+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:00.082894+0000', 'last_peered': '2024-09-11T05:03:00.082894+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:00.082894+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9012, 'log_dups_size': 0, 'ondisk_log_size': 9012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28824736, 'num_objects': 3519, 'num_object_clones': 0, 'num_object_copies': 14076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9010, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1298}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9129", 'reported_seq': 9234, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:00.549345+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:00.549345+0000', 'last_peered': '2024-09-11T05:03:00.549345+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:00.549345+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9129, 'log_dups_size': 0, 'ondisk_log_size': 9129, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28762112, 'num_objects': 3511, 'num_object_clones': 0, 'num_object_copies': 14044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9129, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1217}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9175", 'reported_seq': 9268, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:58.889548+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:02:58.889548+0000', 'last_peered': '2024-09-11T05:02:58.889548+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:02:58.889548+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9175, 'log_dups_size': 0, 'ondisk_log_size': 9175, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29119648, 'num_objects': 3555, 'num_object_clones': 0, 'num_object_copies': 14220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9173, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1248}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9255", 'reported_seq': 14072, 'reported_epoch': 35, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-11T05:03:00.015578+0000', 'last_change': '2024-09-11T05:02:59.080607+0000', 'last_active': '2024-09-11T05:03:00.015578+0000', 'last_peered': '2024-09-11T05:03:00.015578+0000', 'last_clean': '2024-09-11T05:03:00.015578+0000', 'last_became_active': '2024-09-11T05:02:58.942760+0000', 'last_became_peered': '2024-09-11T05:02:58.942760+0000', 'last_unstale': '2024-09-11T05:03:00.015578+0000', 'last_undegraded': '2024-09-11T05:03:00.015578+0000', 'last_fullsized': '2024-09-11T05:03:00.015578+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 35, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 436, 'log_size': 9255, 'log_dups_size': 0, 'ondisk_log_size': 9255, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29242528, 'num_objects': 3570, 'num_object_clones': 0, 'num_object_copies': 14280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9253, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2365, 'num_bytes_recovered': 15564800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9261", 'reported_seq': 9383, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:00.467696+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:00.467696+0000', 'last_peered': '2024-09-11T05:03:00.467696+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:00.467696+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9261, 'log_dups_size': 0, 'ondisk_log_size': 9261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28887360, 'num_objects': 3527, 'num_object_clones': 0, 'num_object_copies': 14108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9258, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9048", 'reported_seq': 9159, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:00.955513+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:00.955513+0000', 'last_peered': '2024-09-11T05:03:00.955513+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:00.955513+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9048, 'log_dups_size': 0, 'ondisk_log_size': 9048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28505248, 'num_objects': 3480, 'num_object_clones': 0, 'num_object_copies': 13920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9046, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1174}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:02.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9119", 'reported_seq': 9235, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:00.359339+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:00.359339+0000', 'last_peered': '2024-09-11T05:03:00.359339+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:00.359339+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9119, 'log_dups_size': 0, 'ondisk_log_size': 9119, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28946160, 'num_objects': 3534, 'num_object_clones': 0, 'num_object_copies': 14136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9116, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1236}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:02.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:02.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9189", 'reported_seq': 9376, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:01.176915+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:01.176915+0000', 'last_peered': '2024-09-11T05:03:01.176915+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:01.176915+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9189, 'log_dups_size': 0, 'ondisk_log_size': 9189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28949072, 'num_objects': 3534, 'num_object_clones': 0, 'num_object_copies': 14136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2282, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9188, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:02.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:02.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9099", 'reported_seq': 9205, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:00.428223+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:00.428223+0000', 'last_peered': '2024-09-11T05:03:00.428223+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:00.428223+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9099, 'log_dups_size': 0, 'ondisk_log_size': 9099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28611744, 'num_objects': 3493, 'num_object_clones': 0, 'num_object_copies': 13972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9097, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1229}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:02.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:02.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9020", 'reported_seq': 9137, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:01.074928+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:01.074928+0000', 'last_peered': '2024-09-11T05:03:01.074928+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:01.074928+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9020, 'log_dups_size': 0, 'ondisk_log_size': 9020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27759776, 'num_objects': 3389, 'num_object_clones': 0, 'num_object_copies': 13556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9018, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:02.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:02.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'9180", 'reported_seq': 9299, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:02:58.888924+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:02:58.888924+0000', 'last_peered': '2024-09-11T05:02:58.888924+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:02:58.888924+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9180, 'log_dups_size': 0, 'ondisk_log_size': 9180, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29463712, 'num_objects': 3597, 'num_object_clones': 0, 'num_object_copies': 14388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9178, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:02.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:02.863 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:02.863+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9284 (0'0,35'9284] local-lis/les=34/35 n=3541 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9284 lcod 35'9283 mlcod 35'9283 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:02.961 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:02.960+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9284 (0'0,35'9284] local-lis/les=34/35 n=3541 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9284 lcod 35'9283 mlcod 35'9283 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:03.139 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:03.133+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9286 (0'0,35'9286] local-lis/les=34/35 n=3539 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9284 crt=35'9284 lcod 35'9283 mlcod 35'9283 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:03.396 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:03.394+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9289 (0'0,35'9289] local-lis/les=34/35 n=3536 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9289 lcod 35'9288 mlcod 35'9288 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:03.649 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31973/224484 objects degraded (14.243%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":510,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56121,"data_bytes":459695752,"bytes_used":2886696960,"bytes_avail":383660359680,"bytes_total":386547056640,"degraded_objects":31973,"degraded_total":224484,"degraded_ratio":0.1424288590723615,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":262},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:03.670 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:03:03.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:04.037 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:04.026+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9302 (0'0,35'9302] local-lis/les=34/35 n=3523 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9298 crt=35'9298 lcod 35'9297 mlcod 35'9297 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:04.210 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:04.209+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9304 (0'0,35'9304] local-lis/les=34/35 n=3521 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9304 lcod 35'9303 mlcod 35'9303 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:04.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:04.347+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9306 (0'0,35'9306] local-lis/les=34/35 n=3519 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9304 crt=35'9304 lcod 35'9303 mlcod 35'9303 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:04.425 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:04.426 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:04.457 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-11T05:03:03.798913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459695752,"num_objects":56121,"num_object_clones":0,"num_object_copies":224484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139072,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2700,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145288,"ondisk_log_size":145288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819040,"kb_used_data":901912,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374668320,"statfs":{"total":386547056640,"available":383660359680,"internally_reserved":0,"allocated":923557888,"data_stored":922805707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1962985014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,11,10,9,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10821632,"num_objects":-1321,"num_object_clones":0,"num_object_copies":-5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1321,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005335"},"pg_stats":[{"pgid":"1.f","version":"35'9036","reported_seq":9130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.091251+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:00.091251+0000","last_peered":"2024-09-11T05:03:00.091251+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:00.091251+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28491776,"num_objects":3478,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'9124","reported_seq":9238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.883247+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:02:58.883247+0000","last_peered":"2024-09-11T05:02:58.883247+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:02:58.883247+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29602976,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8778","reported_seq":8880,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.141294+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:01.141294+0000","last_peered":"2024-09-11T05:03:01.141294+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:01.141294+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8778,"log_dups_size":0,"ondisk_log_size":8778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27703912,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8777,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1172}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'8955","reported_seq":9062,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.881980+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:02:58.881980+0000","last_peered":"2024-09-11T05:02:58.881980+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:02:58.881980+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28349600,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9012","reported_seq":9134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.082894+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:00.082894+0000","last_peered":"2024-09-11T05:03:00.082894+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:00.082894+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9012,"log_dups_size":0,"ondisk_log_size":9012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28824736,"num_objects":3519,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9010,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9129","reported_seq":9234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.549345+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:00.549345+0000","last_peered":"2024-09-11T05:03:00.549345+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:00.549345+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9129,"log_dups_size":0,"ondisk_log_size":9129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28762112,"num_objects":3511,"num_object_clones":0,"num_object_copies":14044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9129,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1217}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9175","reported_seq":9268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.889548+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:02:58.889548+0000","last_peered":"2024-09-11T05:02:58.889548+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:02:58.889548+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119648,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1248}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9255","reported_seq":14072,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T05:03:00.015578+0000","last_change":"2024-09-11T05:02:59.080607+0000","last_active":"2024-09-11T05:03:00.015578+0000","last_peered":"2024-09-11T05:03:00.015578+0000","last_clean":"2024-09-11T05:03:00.015578+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:00.015578+0000","last_undegraded":"2024-09-11T05:03:00.015578+0000","last_fullsized":"2024-09-11T05:03:00.015578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":436,"log_size":9255,"log_dups_size":0,"ondisk_log_size":9255,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29242528,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9253,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2365,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9261","reported_seq":9383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.467696+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:00.467696+0000","last_peered":"2024-09-11T05:03:00.467696+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:00.467696+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9261,"log_dups_size":0,"ondisk_log_size":9261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28887360,"num_objects":3527,"num_object_clones":0,"num_object_copies":14108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9258,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9048","reported_seq":9159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.955513+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:00.955513+0000","last_peered":"2024-09-11T05:03:00.955513+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:00.955513+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9048,"log_dups_size":0,"ondisk_log_size":9048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505248,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1174}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9119","reported_seq":9235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.359339+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:00.359339+0000","last_peered":"2024-09-11T05:03:00.359339+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:00.359339+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9119,"log_dups_size":0,"ondisk_log_size":9119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28946160,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8908","reported_seq":13508,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:00.077126+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:00.077126+0000","last_peered":"2024-09-11T05:03:00.077126+0000","last_clean":"2024-09-11T05:03:00.077126+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:00.077126+0000","last_undegraded":"2024-09-11T05:03:00.077126+0000","last_fullsized":"2024-09-11T05:03:00.077126+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8908,"log_dups_size":0,"ondisk_log_size":8908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28475392,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9189","reported_seq":9376,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.176915+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:01.176915+0000","last_peered":"2024-09-11T05:03:01.176915+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:01.176915+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9189,"log_dups_size":0,"ondisk_log_size":9189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28949072,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9188,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9099","reported_seq":9205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:00.428223+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:00.428223+0000","last_peered":"2024-09-11T05:03:00.428223+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:00.428223+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28611744,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1229}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9020","reported_seq":9137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:01.074928+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:01.074928+0000","last_peered":"2024-09-11T05:03:01.074928+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:01.074928+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9020,"log_dups_size":0,"ondisk_log_size":9020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27759776,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'9180","reported_seq":9299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:02:58.888924+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:02:58.888924+0000","last_peered":"2024-09-11T05:02:58.888924+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:02:58.888924+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9180,"log_dups_size":0,"ondisk_log_size":9180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29463712,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9178,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":459695752,"num_objects":56121,"num_object_clones":0,"num_object_copies":224484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139072,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2700,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":920215552,"data_stored":920215552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145288,"ondisk_log_size":145288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":225256,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":230662144,"data_stored":230472942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733572,"kb_used_data":224804,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93638268,"statfs":{"total":96636764160,"available":95885586432,"internally_reserved":0,"allocated":230199296,"data_stored":230010094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632080,"kb_used_data":225592,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93739760,"statfs":{"total":96636764160,"available":95989514240,"internally_reserved":0,"allocated":231006208,"data_stored":230817006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":416214738},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731312,"kb_used_data":226260,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93640528,"statfs":{"total":96636764160,"available":95887900672,"internally_reserved":0,"allocated":231690240,"data_stored":231505665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,1,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230957056,"data_stored":230957056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230633472,"data_stored":230633472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229036032,"data_stored":229036032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":229588992,"data_stored":229588992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:04.536 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:04.535+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9309 (0'0,35'9309] local-lis/les=34/35 n=3516 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9308 crt=35'9306 lcod 35'9307 mlcod 35'9307 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:04.657 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:04.654+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9311 (0'0,35'9311] local-lis/les=34/35 n=3514 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9309 crt=35'9309 lcod 35'9308 mlcod 35'9308 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:04.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:04.825+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9313 (0'0,35'9313] local-lis/les=34/35 n=3512 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9313 lcod 35'9312 mlcod 35'9312 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:04.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:04.956+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9315 (0'0,35'9315] local-lis/les=34/35 n=3510 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9314 crt=35'9314 lcod 35'9313 mlcod 35'9313 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:05.165 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:05.162+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9316 (0'0,35'9316] local-lis/les=34/35 n=3509 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9315 crt=35'9315 lcod 35'9314 mlcod 35'9314 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:05.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:05.379+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9318 (0'0,35'9318] local-lis/les=34/35 n=3507 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9317 crt=35'9317 lcod 35'9316 mlcod 35'9316 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:05.485 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:05.485+0000 7f1fe53eb640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9320 (0'0,35'9320] local-lis/les=34/35 n=3505 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9318 crt=35'9318 lcod 35'9317 mlcod 35'9317 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:05.608 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:05.606+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9320 (0'0,35'9320] local-lis/les=34/35 n=3505 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 crt=35'9320 lcod 35'9319 mlcod 35'9319 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:05.875 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:05.872+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9322 (0'0,35'9322] local-lis/les=34/35 n=3503 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9321 crt=35'9320 lcod 35'9320 mlcod 35'9320 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:06.178 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:06.178+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9325 (0'0,35'9325] local-lis/les=34/35 n=3500 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9324 crt=35'9323 lcod 35'9323 mlcod 35'9323 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:06.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T05:03:06.249+0000 7f1fe13e3640 -1 osd.1 pg_epoch: 35 pg[1.3s0( v 35'9326 (0'0,35'9326] local-lis/les=34/35 n=3499 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,2,3]p1(0) r=0 lpr=34 luod=35'9325 crt=35'9325 lcod 35'9324 mlcod 35'9324 active+clean+scrubbing [ 1.3s0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-11T05:03:07.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:08.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:08.202 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-11T05:03:07.800554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451782280,"num_objects":55155,"num_object_clones":0,"num_object_copies":220620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140038,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146254,"ondisk_log_size":146254,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801872,"kb_used_data":884780,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374685488,"statfs":{"total":386547056640,"available":383677939712,"internally_reserved":0,"allocated":906014720,"data_stored":905257912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1962985014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,5,7,15,15,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16318464,"num_objects":-1992,"num_object_clones":0,"num_object_copies":-7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.940694"},"pg_stats":[{"pgid":"1.f","version":"35'9091","reported_seq":9185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.429403+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:05.429403+0000","last_peered":"2024-09-11T05:03:05.429403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:05.429403+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28041216,"num_objects":3423,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9091,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9187","reported_seq":9301,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:03.704268+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:03.704268+0000","last_peered":"2024-09-11T05:03:03.704268+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:03.704268+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9187,"log_dups_size":0,"ondisk_log_size":9187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29086880,"num_objects":3551,"num_object_clones":0,"num_object_copies":14204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9185,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8835","reported_seq":8937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.091768+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:06.091768+0000","last_peered":"2024-09-11T05:03:06.091768+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:06.091768+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8835,"log_dups_size":0,"ondisk_log_size":8835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27236968,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8834,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9030","reported_seq":9137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:03.706077+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:03.706077+0000","last_peered":"2024-09-11T05:03:03.706077+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:03.706077+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9030,"log_dups_size":0,"ondisk_log_size":9030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27735200,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9028,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9071","reported_seq":9193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.360021+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:05.360021+0000","last_peered":"2024-09-11T05:03:05.360021+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:05.360021+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28341408,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9069,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9188","reported_seq":9293,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.213662+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:05.213662+0000","last_peered":"2024-09-11T05:03:05.213662+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:05.213662+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9188,"log_dups_size":0,"ondisk_log_size":9188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28278784,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9188,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9227","reported_seq":9320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:03.745786+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:03.745786+0000","last_peered":"2024-09-11T05:03:03.745786+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:03.745786+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28693664,"num_objects":3503,"num_object_clones":0,"num_object_copies":14012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1196}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9317","reported_seq":14163,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T05:03:05.201339+0000","last_change":"2024-09-11T05:02:59.080607+0000","last_active":"2024-09-11T05:03:05.201339+0000","last_peered":"2024-09-11T05:03:05.201339+0000","last_clean":"2024-09-11T05:03:05.201339+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:05.201339+0000","last_undegraded":"2024-09-11T05:03:05.201339+0000","last_fullsized":"2024-09-11T05:03:05.201339+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":2906,"log_size":9317,"log_dups_size":0,"ondisk_log_size":9317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28734624,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9315,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2365,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9316","reported_seq":9438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.495952+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:05.495952+0000","last_peered":"2024-09-11T05:03:05.495952+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:05.495952+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9316,"log_dups_size":0,"ondisk_log_size":9316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28436800,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9313,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9099","reported_seq":9210,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.092971+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:06.092971+0000","last_peered":"2024-09-11T05:03:06.092971+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:06.092971+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9187","reported_seq":9303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.512451+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:05.512451+0000","last_peered":"2024-09-11T05:03:05.512451+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:05.512451+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9187,"log_dups_size":0,"ondisk_log_size":9187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28389104,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8961","reported_seq":13561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:04.798571+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:04.798571+0000","last_peered":"2024-09-11T05:03:04.798571+0000","last_clean":"2024-09-11T05:03:04.798571+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:04.798571+0000","last_undegraded":"2024-09-11T05:03:04.798571+0000","last_fullsized":"2024-09-11T05:03:04.798571+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28041216,"num_objects":3423,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2771,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9250","reported_seq":9517,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.186335+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:06.186335+0000","last_peered":"2024-09-11T05:03:06.186335+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:06.186335+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28449360,"num_objects":3473,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9163","reported_seq":9269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.460256+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:05.460256+0000","last_peered":"2024-09-11T05:03:05.460256+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:05.460256+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9163,"log_dups_size":0,"ondisk_log_size":9163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9161,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9079","reported_seq":9197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.122014+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:06.122014+0000","last_peered":"2024-09-11T05:03:06.122014+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:06.122014+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276448,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9253","reported_seq":9370,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:03.898013+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:03.898013+0000","last_peered":"2024-09-11T05:03:03.898013+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:03.898013+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9253,"log_dups_size":0,"ondisk_log_size":9253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28865696,"num_objects":3524,"num_object_clones":0,"num_object_copies":14096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451782280,"num_objects":55155,"num_object_clones":0,"num_object_copies":220620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140038,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":903565312,"data_stored":903565312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146254,"ondisk_log_size":146254,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722244,"kb_used_data":221264,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93649596,"statfs":{"total":96636764160,"available":95897186304,"internally_reserved":0,"allocated":226574336,"data_stored":226385134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716404,"kb_used_data":220132,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93655436,"statfs":{"total":96636764160,"available":95903166464,"internally_reserved":0,"allocated":225415168,"data_stored":225225966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,0,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631720,"kb_used_data":221080,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93740120,"statfs":{"total":96636764160,"available":95989882880,"internally_reserved":0,"allocated":226385920,"data_stored":226196718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420474578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731504,"kb_used_data":222304,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93640336,"statfs":{"total":96636764160,"available":95887704064,"internally_reserved":0,"allocated":227639296,"data_stored":227450094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226856960,"data_stored":226856960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226054144,"data_stored":226054144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225091584,"data_stored":225091584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225562624,"data_stored":225562624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:08.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:08.839 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:08.840 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:08.880 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-11T05:03:07.800554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451782280,"num_objects":55155,"num_object_clones":0,"num_object_copies":220620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140038,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146254,"ondisk_log_size":146254,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801872,"kb_used_data":884780,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374685488,"statfs":{"total":386547056640,"available":383677939712,"internally_reserved":0,"allocated":906014720,"data_stored":905257912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1962985014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,5,7,15,15,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16318464,"num_objects":-1992,"num_object_clones":0,"num_object_copies":-7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.940694"},"pg_stats":[{"pgid":"1.f","version":"35'9091","reported_seq":9185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.429403+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:05.429403+0000","last_peered":"2024-09-11T05:03:05.429403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:05.429403+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28041216,"num_objects":3423,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9091,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9187","reported_seq":9301,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:03.704268+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:03.704268+0000","last_peered":"2024-09-11T05:03:03.704268+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:03.704268+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9187,"log_dups_size":0,"ondisk_log_size":9187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29086880,"num_objects":3551,"num_object_clones":0,"num_object_copies":14204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9185,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8835","reported_seq":8937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.091768+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:06.091768+0000","last_peered":"2024-09-11T05:03:06.091768+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:06.091768+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8835,"log_dups_size":0,"ondisk_log_size":8835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27236968,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8834,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9030","reported_seq":9137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:03.706077+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:03.706077+0000","last_peered":"2024-09-11T05:03:03.706077+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:03.706077+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9030,"log_dups_size":0,"ondisk_log_size":9030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27735200,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9028,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9071","reported_seq":9193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.360021+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:05.360021+0000","last_peered":"2024-09-11T05:03:05.360021+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:05.360021+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28341408,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9069,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9188","reported_seq":9293,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.213662+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:05.213662+0000","last_peered":"2024-09-11T05:03:05.213662+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:05.213662+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9188,"log_dups_size":0,"ondisk_log_size":9188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28278784,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9188,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9227","reported_seq":9320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:03.745786+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:03.745786+0000","last_peered":"2024-09-11T05:03:03.745786+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:03.745786+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28693664,"num_objects":3503,"num_object_clones":0,"num_object_copies":14012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1196}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9317","reported_seq":14163,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T05:03:05.201339+0000","last_change":"2024-09-11T05:02:59.080607+0000","last_active":"2024-09-11T05:03:05.201339+0000","last_peered":"2024-09-11T05:03:05.201339+0000","last_clean":"2024-09-11T05:03:05.201339+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:05.201339+0000","last_undegraded":"2024-09-11T05:03:05.201339+0000","last_fullsized":"2024-09-11T05:03:05.201339+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":2906,"log_size":9317,"log_dups_size":0,"ondisk_log_size":9317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28734624,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9315,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2365,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9316","reported_seq":9438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.495952+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:05.495952+0000","last_peered":"2024-09-11T05:03:05.495952+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:05.495952+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9316,"log_dups_size":0,"ondisk_log_size":9316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28436800,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9313,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9099","reported_seq":9210,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.092971+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:06.092971+0000","last_peered":"2024-09-11T05:03:06.092971+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:06.092971+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9187","reported_seq":9303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.512451+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:05.512451+0000","last_peered":"2024-09-11T05:03:05.512451+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:05.512451+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9187,"log_dups_size":0,"ondisk_log_size":9187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28389104,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8961","reported_seq":13561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:04.798571+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:04.798571+0000","last_peered":"2024-09-11T05:03:04.798571+0000","last_clean":"2024-09-11T05:03:04.798571+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:04.798571+0000","last_undegraded":"2024-09-11T05:03:04.798571+0000","last_fullsized":"2024-09-11T05:03:04.798571+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28041216,"num_objects":3423,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2771,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9250","reported_seq":9517,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.186335+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:06.186335+0000","last_peered":"2024-09-11T05:03:06.186335+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:06.186335+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28449360,"num_objects":3473,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9163","reported_seq":9269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.460256+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:05.460256+0000","last_peered":"2024-09-11T05:03:05.460256+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:05.460256+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9163,"log_dups_size":0,"ondisk_log_size":9163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9161,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9079","reported_seq":9197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.122014+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:06.122014+0000","last_peered":"2024-09-11T05:03:06.122014+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:06.122014+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276448,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9253","reported_seq":9370,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:03.898013+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:03.898013+0000","last_peered":"2024-09-11T05:03:03.898013+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:03.898013+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9253,"log_dups_size":0,"ondisk_log_size":9253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28865696,"num_objects":3524,"num_object_clones":0,"num_object_copies":14096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451782280,"num_objects":55155,"num_object_clones":0,"num_object_copies":220620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140038,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":903565312,"data_stored":903565312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146254,"ondisk_log_size":146254,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722244,"kb_used_data":221264,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93649596,"statfs":{"total":96636764160,"available":95897186304,"internally_reserved":0,"allocated":226574336,"data_stored":226385134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716404,"kb_used_data":220132,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93655436,"statfs":{"total":96636764160,"available":95903166464,"internally_reserved":0,"allocated":225415168,"data_stored":225225966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,0,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631720,"kb_used_data":221080,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93740120,"statfs":{"total":96636764160,"available":95989882880,"internally_reserved":0,"allocated":226385920,"data_stored":226196718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420474578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731504,"kb_used_data":222304,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93640336,"statfs":{"total":96636764160,"available":95887704064,"internally_reserved":0,"allocated":227639296,"data_stored":227450094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226856960,"data_stored":226856960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226054144,"data_stored":226054144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225091584,"data_stored":225091584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225562624,"data_stored":225562624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:08.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:08.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9091", 'reported_seq': 9185, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:05.429403+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:05.429403+0000', 'last_peered': '2024-09-11T05:03:05.429403+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:05.429403+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9091, 'log_dups_size': 0, 'ondisk_log_size': 9091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28041216, 'num_objects': 3423, 'num_object_clones': 0, 'num_object_copies': 13692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3423, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9091, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:08.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:08.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9187", 'reported_seq': 9301, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:03.704268+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:03.704268+0000', 'last_peered': '2024-09-11T05:03:03.704268+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:03.704268+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9187, 'log_dups_size': 0, 'ondisk_log_size': 9187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29086880, 'num_objects': 3551, 'num_object_clones': 0, 'num_object_copies': 14204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3551, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9185, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:08.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:08.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8835", 'reported_seq': 8937, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:06.091768+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:06.091768+0000', 'last_peered': '2024-09-11T05:03:06.091768+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:06.091768+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8835, 'log_dups_size': 0, 'ondisk_log_size': 8835, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27236968, 'num_objects': 3326, 'num_object_clones': 0, 'num_object_copies': 13304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3326, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8834, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1115}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9030", 'reported_seq': 9137, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:03.706077+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:03.706077+0000', 'last_peered': '2024-09-11T05:03:03.706077+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:03.706077+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9030, 'log_dups_size': 0, 'ondisk_log_size': 9030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27735200, 'num_objects': 3386, 'num_object_clones': 0, 'num_object_copies': 13544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3386, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9028, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9071", 'reported_seq': 9193, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:05.360021+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:05.360021+0000', 'last_peered': '2024-09-11T05:03:05.360021+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:05.360021+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9071, 'log_dups_size': 0, 'ondisk_log_size': 9071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28341408, 'num_objects': 3460, 'num_object_clones': 0, 'num_object_copies': 13840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9069, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1239}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9188", 'reported_seq': 9293, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:05.213662+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:05.213662+0000', 'last_peered': '2024-09-11T05:03:05.213662+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:05.213662+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9188, 'log_dups_size': 0, 'ondisk_log_size': 9188, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28278784, 'num_objects': 3452, 'num_object_clones': 0, 'num_object_copies': 13808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9188, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1158}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9227", 'reported_seq': 9320, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:03.745786+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:03.745786+0000', 'last_peered': '2024-09-11T05:03:03.745786+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:03.745786+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9227, 'log_dups_size': 0, 'ondisk_log_size': 9227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28693664, 'num_objects': 3503, 'num_object_clones': 0, 'num_object_copies': 14012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9225, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1196}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:08.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9317", 'reported_seq': 14163, 'reported_epoch': 35, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-11T05:03:05.201339+0000', 'last_change': '2024-09-11T05:02:59.080607+0000', 'last_active': '2024-09-11T05:03:05.201339+0000', 'last_peered': '2024-09-11T05:03:05.201339+0000', 'last_clean': '2024-09-11T05:03:05.201339+0000', 'last_became_active': '2024-09-11T05:02:58.942760+0000', 'last_became_peered': '2024-09-11T05:02:58.942760+0000', 'last_unstale': '2024-09-11T05:03:05.201339+0000', 'last_undegraded': '2024-09-11T05:03:05.201339+0000', 'last_fullsized': '2024-09-11T05:03:05.201339+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 35, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 2906, 'log_size': 9317, 'log_dups_size': 0, 'ondisk_log_size': 9317, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 6s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28734624, 'num_objects': 3508, 'num_object_clones': 0, 'num_object_copies': 14032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9315, 'num_write_kb': 51292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2365, 'num_bytes_recovered': 15564800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9316", 'reported_seq': 9438, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:05.495952+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:05.495952+0000', 'last_peered': '2024-09-11T05:03:05.495952+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:05.495952+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9316, 'log_dups_size': 0, 'ondisk_log_size': 9316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28436800, 'num_objects': 3472, 'num_object_clones': 0, 'num_object_copies': 13888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9313, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1106}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9099", 'reported_seq': 9210, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:06.092971+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:06.092971+0000', 'last_peered': '2024-09-11T05:03:06.092971+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:06.092971+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9099, 'log_dups_size': 0, 'ondisk_log_size': 9099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28087456, 'num_objects': 3429, 'num_object_clones': 0, 'num_object_copies': 13716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9097, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1123}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9187", 'reported_seq': 9303, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:05.512451+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:05.512451+0000', 'last_peered': '2024-09-11T05:03:05.512451+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:05.512451+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9187, 'log_dups_size': 0, 'ondisk_log_size': 9187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28389104, 'num_objects': 3466, 'num_object_clones': 0, 'num_object_copies': 13864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9184, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1168}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9250", 'reported_seq': 9517, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:06.186335+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:06.186335+0000', 'last_peered': '2024-09-11T05:03:06.186335+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:06.186335+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9250, 'log_dups_size': 0, 'ondisk_log_size': 9250, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28449360, 'num_objects': 3473, 'num_object_clones': 0, 'num_object_copies': 13892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9249, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 92, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:08.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9163", 'reported_seq': 9269, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:05.460256+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:05.460256+0000', 'last_peered': '2024-09-11T05:03:05.460256+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:05.460256+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9163, 'log_dups_size': 0, 'ondisk_log_size': 9163, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28087456, 'num_objects': 3429, 'num_object_clones': 0, 'num_object_copies': 13716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9161, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1165}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:08.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:08.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9079", 'reported_seq': 9197, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:06.122014+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:06.122014+0000', 'last_peered': '2024-09-11T05:03:06.122014+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:06.122014+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9079, 'log_dups_size': 0, 'ondisk_log_size': 9079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27276448, 'num_objects': 3330, 'num_object_clones': 0, 'num_object_copies': 13320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9077, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:08.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:08.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9253", 'reported_seq': 9370, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:03.898013+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:03.898013+0000', 'last_peered': '2024-09-11T05:03:03.898013+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:03.898013+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9253, 'log_dups_size': 0, 'ondisk_log_size': 9253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28865696, 'num_objects': 3524, 'num_object_clones': 0, 'num_object_copies': 14096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9251, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1236}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:08.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:09.669 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:09.670 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31933/220620 objects degraded (14.474%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":516,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55155,"data_bytes":451782280,"bytes_used":2869116928,"bytes_avail":383677939712,"bytes_total":386547056640,"degraded_objects":31933,"degraded_total":220620,"degraded_ratio":0.14474209047230532,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:09.692 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:03:09.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:10.357 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:10.376 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-11T05:03:09.801474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449668744,"num_objects":54897,"num_object_clones":0,"num_object_copies":219588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54897,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140296,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146512,"ondisk_log_size":146512,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785584,"kb_used_data":880952,"kb_used_omap":14,"kb_used_meta":1904497,"kb_avail":374701776,"statfs":{"total":386547056640,"available":383694618624,"internally_reserved":0,"allocated":902094848,"data_stored":901338040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1950205494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,5,8,9,19,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18432000,"num_objects":-2250,"num_object_clones":0,"num_object_copies":-9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.936603"},"pg_stats":[{"pgid":"1.f","version":"35'9091","reported_seq":9185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.429403+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:05.429403+0000","last_peered":"2024-09-11T05:03:05.429403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:05.429403+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28041216,"num_objects":3423,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9091,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9262","reported_seq":9376,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.856806+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:08.856806+0000","last_peered":"2024-09-11T05:03:08.856806+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:08.856806+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9262,"log_dups_size":0,"ondisk_log_size":9262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28472480,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9260,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8835","reported_seq":8937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.091768+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:06.091768+0000","last_peered":"2024-09-11T05:03:06.091768+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:06.091768+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8835,"log_dups_size":0,"ondisk_log_size":8835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27236968,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8834,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9097","reported_seq":9204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.866295+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:08.866295+0000","last_peered":"2024-09-11T05:03:08.866295+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:08.866295+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27186336,"num_objects":3319,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9071","reported_seq":9193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.360021+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:05.360021+0000","last_peered":"2024-09-11T05:03:05.360021+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:05.360021+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28341408,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9069,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9188","reported_seq":9293,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.213662+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:05.213662+0000","last_peered":"2024-09-11T05:03:05.213662+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:05.213662+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9188,"log_dups_size":0,"ondisk_log_size":9188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28278784,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9188,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9291","reported_seq":9383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.936670+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:08.936670+0000","last_peered":"2024-09-11T05:03:08.936670+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:08.936670+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28169376,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9289,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1132}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9317","reported_seq":14163,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T05:03:05.201339+0000","last_change":"2024-09-11T05:02:59.080607+0000","last_active":"2024-09-11T05:03:05.201339+0000","last_peered":"2024-09-11T05:03:05.201339+0000","last_clean":"2024-09-11T05:03:05.201339+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:05.201339+0000","last_undegraded":"2024-09-11T05:03:05.201339+0000","last_fullsized":"2024-09-11T05:03:05.201339+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":2906,"log_size":9317,"log_dups_size":0,"ondisk_log_size":9317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28734624,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9315,"num_write_kb":51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2365,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9316","reported_seq":9438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.495952+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:05.495952+0000","last_peered":"2024-09-11T05:03:05.495952+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:05.495952+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9316,"log_dups_size":0,"ondisk_log_size":9316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28436800,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9313,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9099","reported_seq":9210,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.092971+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:06.092971+0000","last_peered":"2024-09-11T05:03:06.092971+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:06.092971+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9187","reported_seq":9303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.512451+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:05.512451+0000","last_peered":"2024-09-11T05:03:05.512451+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:05.512451+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9187,"log_dups_size":0,"ondisk_log_size":9187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28389104,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8961","reported_seq":13561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:04.798571+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:04.798571+0000","last_peered":"2024-09-11T05:03:04.798571+0000","last_clean":"2024-09-11T05:03:04.798571+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:04.798571+0000","last_undegraded":"2024-09-11T05:03:04.798571+0000","last_fullsized":"2024-09-11T05:03:04.798571+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28041216,"num_objects":3423,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2771,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9250","reported_seq":9517,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.186335+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:06.186335+0000","last_peered":"2024-09-11T05:03:06.186335+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:06.186335+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28449360,"num_objects":3473,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9163","reported_seq":9269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:05.460256+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:05.460256+0000","last_peered":"2024-09-11T05:03:05.460256+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:05.460256+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9163,"log_dups_size":0,"ondisk_log_size":9163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28087456,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9161,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9079","reported_seq":9197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:06.122014+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:06.122014+0000","last_peered":"2024-09-11T05:03:06.122014+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:06.122014+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276448,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9305","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.680317+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:08.680317+0000","last_peered":"2024-09-11T05:03:08.680317+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:08.680317+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9305,"log_dups_size":0,"ondisk_log_size":9305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28439712,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9303,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":449668744,"num_objects":54897,"num_object_clones":0,"num_object_copies":219588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54897,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140296,"num_write_kb":756004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":899629056,"data_stored":899629056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146512,"ondisk_log_size":146512,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722244,"kb_used_data":221264,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93649596,"statfs":{"total":96636764160,"available":95897186304,"internally_reserved":0,"allocated":226574336,"data_stored":226385134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716404,"kb_used_data":220132,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93655436,"statfs":{"total":96636764160,"available":95903166464,"internally_reserved":0,"allocated":225415168,"data_stored":225225966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,0,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631720,"kb_used_data":221080,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93740120,"statfs":{"total":96636764160,"available":95989882880,"internally_reserved":0,"allocated":226385920,"data_stored":226196718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420474578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715216,"kb_used_data":218476,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93656624,"statfs":{"total":96636764160,"available":95904382976,"internally_reserved":0,"allocated":223719424,"data_stored":223530222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":222920704,"data_stored":222920704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226054144,"data_stored":226054144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225091584,"data_stored":225091584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225562624,"data_stored":225562624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:13.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:13.991 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:14.011 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-11T05:03:13.802896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":443683304,"num_objects":54166,"num_object_clones":0,"num_object_copies":216664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131737,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147241,"ondisk_log_size":147241,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785556,"kb_used_data":868468,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374701804,"statfs":{"total":386547056640,"available":383694647296,"internally_reserved":0,"allocated":889311232,"data_stored":888554424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1962985014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,3,17,22,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16012448,"num_objects":-1955,"num_object_clones":0,"num_object_copies":-7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7335,"num_write_kb":-51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2269,"num_bytes_recovered":-15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007060"},"pg_stats":[{"pgid":"1.f","version":"35'9159","reported_seq":9253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.536838+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:10.536838+0000","last_peered":"2024-09-11T05:03:10.536838+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:10.536838+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9159,"log_dups_size":0,"ondisk_log_size":9159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27484160,"num_objects":3355,"num_object_clones":0,"num_object_copies":13420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9262","reported_seq":9376,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.856806+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:08.856806+0000","last_peered":"2024-09-11T05:03:08.856806+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:08.856806+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9262,"log_dups_size":0,"ondisk_log_size":9262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28472480,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9260,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8888","reported_seq":8992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.205725+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:11.205725+0000","last_peered":"2024-09-11T05:03:11.205725+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:11.205725+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8888,"log_dups_size":0,"ondisk_log_size":8888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26802792,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8887,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9097","reported_seq":9204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.866295+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:08.866295+0000","last_peered":"2024-09-11T05:03:08.866295+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:08.866295+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27186336,"num_objects":3319,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9130","reported_seq":9252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.644121+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:10.644121+0000","last_peered":"2024-09-11T05:03:10.644121+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:10.644121+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9130,"log_dups_size":0,"ondisk_log_size":9130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27858080,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9128,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9250","reported_seq":9355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.417850+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:10.417850+0000","last_peered":"2024-09-11T05:03:10.417850+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:10.417850+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27770880,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9291","reported_seq":9383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.936670+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:08.936670+0000","last_peered":"2024-09-11T05:03:08.936670+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:08.936670+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28169376,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9289,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1132}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9374","reported_seq":14231,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:10.240949+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:10.240949+0000","last_peered":"2024-09-11T05:03:10.240949+0000","last_clean":"2024-09-11T05:03:10.240949+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:10.240949+0000","last_undegraded":"2024-09-11T05:03:10.240949+0000","last_fullsized":"2024-09-11T05:03:10.240949+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9374,"log_dups_size":0,"ondisk_log_size":9374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28254208,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9375","reported_seq":9497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.346720+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:10.346720+0000","last_peered":"2024-09-11T05:03:10.346720+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:10.346720+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27953472,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9372,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9160","reported_seq":9271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.265427+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:11.265427+0000","last_peered":"2024-09-11T05:03:11.265427+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:11.265427+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27587744,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9251","reported_seq":9367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.489901+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:10.489901+0000","last_peered":"2024-09-11T05:03:10.489901+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:10.489901+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27864816,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9248,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9012","reported_seq":13612,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:10.395148+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:10.395148+0000","last_peered":"2024-09-11T05:03:10.395148+0000","last_clean":"2024-09-11T05:03:10.395148+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:10.395148+0000","last_undegraded":"2024-09-11T05:03:10.395148+0000","last_fullsized":"2024-09-11T05:03:10.395148+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9012,"log_dups_size":0,"ondisk_log_size":9012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27623424,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2822,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9321","reported_seq":9700,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.241370+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:11.241370+0000","last_peered":"2024-09-11T05:03:11.241370+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:11.241370+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27867728,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9320,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9220","reported_seq":9326,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.544674+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:10.544674+0000","last_peered":"2024-09-11T05:03:10.544674+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:10.544674+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9220,"log_dups_size":0,"ondisk_log_size":9220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620512,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9218,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9146","reported_seq":9264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.126009+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:11.126009+0000","last_peered":"2024-09-11T05:03:11.126009+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:11.126009+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9146,"log_dups_size":0,"ondisk_log_size":9146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727584,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9305","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.680317+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:08.680317+0000","last_peered":"2024-09-11T05:03:08.680317+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:08.680317+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9305,"log_dups_size":0,"ondisk_log_size":9305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28439712,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9303,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":443683304,"num_objects":54166,"num_object_clones":0,"num_object_copies":216664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131737,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":887304192,"data_stored":887304192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147241,"ondisk_log_size":147241,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721804,"kb_used_data":216700,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93650036,"statfs":{"total":96636764160,"available":95897636864,"internally_reserved":0,"allocated":221900800,"data_stored":221711598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716692,"kb_used_data":216260,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93655148,"statfs":{"total":96636764160,"available":95902871552,"internally_reserved":0,"allocated":221450240,"data_stored":221261038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,5,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631844,"kb_used_data":217032,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93739996,"statfs":{"total":96636764160,"available":95989755904,"internally_reserved":0,"allocated":222240768,"data_stored":222051566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424734418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715216,"kb_used_data":218476,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93656624,"statfs":{"total":96636764160,"available":95904382976,"internally_reserved":0,"allocated":223719424,"data_stored":223530222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":222920704,"data_stored":222920704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221818880,"data_stored":221818880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220983296,"data_stored":220983296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221581312,"data_stored":221581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:14.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:14.667 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:14.668 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:14.687 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-11T05:03:13.802896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":443683304,"num_objects":54166,"num_object_clones":0,"num_object_copies":216664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131737,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147241,"ondisk_log_size":147241,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785556,"kb_used_data":868468,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374701804,"statfs":{"total":386547056640,"available":383694647296,"internally_reserved":0,"allocated":889311232,"data_stored":888554424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1962985014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,3,17,22,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16012448,"num_objects":-1955,"num_object_clones":0,"num_object_copies":-7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7335,"num_write_kb":-51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2269,"num_bytes_recovered":-15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007060"},"pg_stats":[{"pgid":"1.f","version":"35'9159","reported_seq":9253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.536838+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:10.536838+0000","last_peered":"2024-09-11T05:03:10.536838+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:10.536838+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9159,"log_dups_size":0,"ondisk_log_size":9159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27484160,"num_objects":3355,"num_object_clones":0,"num_object_copies":13420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9262","reported_seq":9376,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.856806+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:08.856806+0000","last_peered":"2024-09-11T05:03:08.856806+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:08.856806+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9262,"log_dups_size":0,"ondisk_log_size":9262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28472480,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9260,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8888","reported_seq":8992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.205725+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:11.205725+0000","last_peered":"2024-09-11T05:03:11.205725+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:11.205725+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8888,"log_dups_size":0,"ondisk_log_size":8888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26802792,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8887,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9097","reported_seq":9204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.866295+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:08.866295+0000","last_peered":"2024-09-11T05:03:08.866295+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:08.866295+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27186336,"num_objects":3319,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9130","reported_seq":9252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.644121+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:10.644121+0000","last_peered":"2024-09-11T05:03:10.644121+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:10.644121+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9130,"log_dups_size":0,"ondisk_log_size":9130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27858080,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9128,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9250","reported_seq":9355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.417850+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:10.417850+0000","last_peered":"2024-09-11T05:03:10.417850+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:10.417850+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27770880,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9291","reported_seq":9383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.936670+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:08.936670+0000","last_peered":"2024-09-11T05:03:08.936670+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:08.936670+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28169376,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9289,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1132}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9374","reported_seq":14231,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:10.240949+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:10.240949+0000","last_peered":"2024-09-11T05:03:10.240949+0000","last_clean":"2024-09-11T05:03:10.240949+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:10.240949+0000","last_undegraded":"2024-09-11T05:03:10.240949+0000","last_fullsized":"2024-09-11T05:03:10.240949+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9374,"log_dups_size":0,"ondisk_log_size":9374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28254208,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9375","reported_seq":9497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.346720+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:10.346720+0000","last_peered":"2024-09-11T05:03:10.346720+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:10.346720+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27953472,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9372,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9160","reported_seq":9271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.265427+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:11.265427+0000","last_peered":"2024-09-11T05:03:11.265427+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:11.265427+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27587744,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9251","reported_seq":9367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.489901+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:10.489901+0000","last_peered":"2024-09-11T05:03:10.489901+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:10.489901+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27864816,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9248,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9012","reported_seq":13612,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:10.395148+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:10.395148+0000","last_peered":"2024-09-11T05:03:10.395148+0000","last_clean":"2024-09-11T05:03:10.395148+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:10.395148+0000","last_undegraded":"2024-09-11T05:03:10.395148+0000","last_fullsized":"2024-09-11T05:03:10.395148+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9012,"log_dups_size":0,"ondisk_log_size":9012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27623424,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2822,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9321","reported_seq":9700,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.241370+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:11.241370+0000","last_peered":"2024-09-11T05:03:11.241370+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:11.241370+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27867728,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9320,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9220","reported_seq":9326,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.544674+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:10.544674+0000","last_peered":"2024-09-11T05:03:10.544674+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:10.544674+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9220,"log_dups_size":0,"ondisk_log_size":9220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620512,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9218,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9146","reported_seq":9264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.126009+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:11.126009+0000","last_peered":"2024-09-11T05:03:11.126009+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:11.126009+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9146,"log_dups_size":0,"ondisk_log_size":9146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727584,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9305","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:08.680317+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:08.680317+0000","last_peered":"2024-09-11T05:03:08.680317+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:08.680317+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9305,"log_dups_size":0,"ondisk_log_size":9305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28439712,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9303,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":443683304,"num_objects":54166,"num_object_clones":0,"num_object_copies":216664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131737,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":887304192,"data_stored":887304192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147241,"ondisk_log_size":147241,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721804,"kb_used_data":216700,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93650036,"statfs":{"total":96636764160,"available":95897636864,"internally_reserved":0,"allocated":221900800,"data_stored":221711598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716692,"kb_used_data":216260,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93655148,"statfs":{"total":96636764160,"available":95902871552,"internally_reserved":0,"allocated":221450240,"data_stored":221261038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,5,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631844,"kb_used_data":217032,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93739996,"statfs":{"total":96636764160,"available":95989755904,"internally_reserved":0,"allocated":222240768,"data_stored":222051566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424734418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715216,"kb_used_data":218476,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93656624,"statfs":{"total":96636764160,"available":95904382976,"internally_reserved":0,"allocated":223719424,"data_stored":223530222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":222920704,"data_stored":222920704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221818880,"data_stored":221818880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220983296,"data_stored":220983296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221581312,"data_stored":221581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:14.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:14.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9159", 'reported_seq': 9253, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:10.536838+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:10.536838+0000', 'last_peered': '2024-09-11T05:03:10.536838+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:10.536838+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9159, 'log_dups_size': 0, 'ondisk_log_size': 9159, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27484160, 'num_objects': 3355, 'num_object_clones': 0, 'num_object_copies': 13420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9159, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1064}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:14.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:14.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9262", 'reported_seq': 9376, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:08.856806+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:08.856806+0000', 'last_peered': '2024-09-11T05:03:08.856806+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:08.856806+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9262, 'log_dups_size': 0, 'ondisk_log_size': 9262, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28472480, 'num_objects': 3476, 'num_object_clones': 0, 'num_object_copies': 13904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9260, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:14.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8888", 'reported_seq': 8992, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:11.205725+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:11.205725+0000', 'last_peered': '2024-09-11T05:03:11.205725+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:11.205725+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8888, 'log_dups_size': 0, 'ondisk_log_size': 8888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26802792, 'num_objects': 3273, 'num_object_clones': 0, 'num_object_copies': 13092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3273, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8887, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1062}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9097", 'reported_seq': 9204, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:08.866295+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:08.866295+0000', 'last_peered': '2024-09-11T05:03:08.866295+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:08.866295+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9097, 'log_dups_size': 0, 'ondisk_log_size': 9097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27186336, 'num_objects': 3319, 'num_object_clones': 0, 'num_object_copies': 13276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9095, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9130", 'reported_seq': 9252, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:10.644121+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:10.644121+0000', 'last_peered': '2024-09-11T05:03:10.644121+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:10.644121+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9130, 'log_dups_size': 0, 'ondisk_log_size': 9130, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27858080, 'num_objects': 3401, 'num_object_clones': 0, 'num_object_copies': 13604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9128, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1180}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9250", 'reported_seq': 9355, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:10.417850+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:10.417850+0000', 'last_peered': '2024-09-11T05:03:10.417850+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:10.417850+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9250, 'log_dups_size': 0, 'ondisk_log_size': 9250, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27770880, 'num_objects': 3390, 'num_object_clones': 0, 'num_object_copies': 13560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9250, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1096}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9291", 'reported_seq': 9383, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:08.936670+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:08.936670+0000', 'last_peered': '2024-09-11T05:03:08.936670+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:08.936670+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9291, 'log_dups_size': 0, 'ondisk_log_size': 9291, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28169376, 'num_objects': 3439, 'num_object_clones': 0, 'num_object_copies': 13756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9289, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1132}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:14.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9375", 'reported_seq': 9497, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:10.346720+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:10.346720+0000', 'last_peered': '2024-09-11T05:03:10.346720+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:10.346720+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9375, 'log_dups_size': 0, 'ondisk_log_size': 9375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27953472, 'num_objects': 3413, 'num_object_clones': 0, 'num_object_copies': 13652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9372, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9160", 'reported_seq': 9271, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:11.265427+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:11.265427+0000', 'last_peered': '2024-09-11T05:03:11.265427+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:11.265427+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9160, 'log_dups_size': 0, 'ondisk_log_size': 9160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27587744, 'num_objects': 3368, 'num_object_clones': 0, 'num_object_copies': 13472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9158, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1062}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9251", 'reported_seq': 9367, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:10.489901+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:10.489901+0000', 'last_peered': '2024-09-11T05:03:10.489901+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:10.489901+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9251, 'log_dups_size': 0, 'ondisk_log_size': 9251, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27864816, 'num_objects': 3402, 'num_object_clones': 0, 'num_object_copies': 13608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9248, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1104}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9321", 'reported_seq': 9700, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:11.241370+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:11.241370+0000', 'last_peered': '2024-09-11T05:03:11.241370+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:11.241370+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9321, 'log_dups_size': 0, 'ondisk_log_size': 9321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27867728, 'num_objects': 3402, 'num_object_clones': 0, 'num_object_copies': 13608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9320, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 148, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9220", 'reported_seq': 9326, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:10.544674+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:10.544674+0000', 'last_peered': '2024-09-11T05:03:10.544674+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:10.544674+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9220, 'log_dups_size': 0, 'ondisk_log_size': 9220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27620512, 'num_objects': 3372, 'num_object_clones': 0, 'num_object_copies': 13488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9218, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9146", 'reported_seq': 9264, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:11.126009+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:11.126009+0000', 'last_peered': '2024-09-11T05:03:11.126009+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:11.126009+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9146, 'log_dups_size': 0, 'ondisk_log_size': 9146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26727584, 'num_objects': 3263, 'num_object_clones': 0, 'num_object_copies': 13052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9144, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:14.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:14.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9305", 'reported_seq': 9424, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:08.680317+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:08.680317+0000', 'last_peered': '2024-09-11T05:03:08.680317+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:08.680317+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9305, 'log_dups_size': 0, 'ondisk_log_size': 9305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28439712, 'num_objects': 3472, 'num_object_clones': 0, 'num_object_copies': 13888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9303, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:14.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:15.458 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:15.459 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31877/216664 objects degraded (14.713%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":522,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54166,"data_bytes":443683304,"bytes_used":2852409344,"bytes_avail":383694647296,"bytes_total":386547056640,"degraded_objects":31877,"degraded_total":216664,"degraded_ratio":0.14712642617139904},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:15.481 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:03:15.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:16.095 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:16.095 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:16.123 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-11T05:03:15.803764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":440283624,"num_objects":53751,"num_object_clones":0,"num_object_copies":215004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53751,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132152,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147656,"ondisk_log_size":147656,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785268,"kb_used_data":859872,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374702092,"statfs":{"total":386547056640,"available":383694942208,"internally_reserved":0,"allocated":880508928,"data_stored":879752120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1971504694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,2,2,15,16,20,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19412128,"num_objects":-2370,"num_object_clones":0,"num_object_copies":-9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6920,"num_write_kb":-51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2269,"num_bytes_recovered":-15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004851"},"pg_stats":[{"pgid":"1.f","version":"35'9159","reported_seq":9253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.536838+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:10.536838+0000","last_peered":"2024-09-11T05:03:10.536838+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:10.536838+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9159,"log_dups_size":0,"ondisk_log_size":9159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27484160,"num_objects":3355,"num_object_clones":0,"num_object_copies":13420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9321","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:13.891868+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:13.891868+0000","last_peered":"2024-09-11T05:03:13.891868+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:13.891868+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27989152,"num_objects":3417,"num_object_clones":0,"num_object_copies":13668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8888","reported_seq":8992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.205725+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:11.205725+0000","last_peered":"2024-09-11T05:03:11.205725+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:11.205725+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8888,"log_dups_size":0,"ondisk_log_size":8888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26802792,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8887,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9154","reported_seq":9261,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:13.721265+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:13.721265+0000","last_peered":"2024-09-11T05:03:13.721265+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:13.721265+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719392,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9130","reported_seq":9252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.644121+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:10.644121+0000","last_peered":"2024-09-11T05:03:10.644121+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:10.644121+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9130,"log_dups_size":0,"ondisk_log_size":9130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27858080,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9128,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9250","reported_seq":9355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.417850+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:10.417850+0000","last_peered":"2024-09-11T05:03:10.417850+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:10.417850+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27770880,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9366","reported_seq":9459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:13.882511+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:13.882511+0000","last_peered":"2024-09-11T05:03:13.882511+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:13.882511+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27554976,"num_objects":3364,"num_object_clones":0,"num_object_copies":13456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1057}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9444","reported_seq":14301,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:15.425442+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:15.425442+0000","last_peered":"2024-09-11T05:03:15.425442+0000","last_clean":"2024-09-11T05:03:15.425442+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:15.425442+0000","last_undegraded":"2024-09-11T05:03:15.425442+0000","last_fullsized":"2024-09-11T05:03:15.425442+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9444,"log_dups_size":0,"ondisk_log_size":9444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27680768,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9375","reported_seq":9497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.346720+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:10.346720+0000","last_peered":"2024-09-11T05:03:10.346720+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:10.346720+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27953472,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9372,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9160","reported_seq":9271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.265427+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:11.265427+0000","last_peered":"2024-09-11T05:03:11.265427+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:11.265427+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27587744,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1062}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9251","reported_seq":9367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.489901+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:10.489901+0000","last_peered":"2024-09-11T05:03:10.489901+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:10.489901+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27864816,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9248,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9086","reported_seq":13685,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:15.487092+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:15.487092+0000","last_peered":"2024-09-11T05:03:15.487092+0000","last_clean":"2024-09-11T05:03:15.487092+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:15.487092+0000","last_undegraded":"2024-09-11T05:03:15.487092+0000","last_fullsized":"2024-09-11T05:03:15.487092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9086,"log_dups_size":0,"ondisk_log_size":9086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27017216,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9321","reported_seq":9700,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.241370+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:11.241370+0000","last_peered":"2024-09-11T05:03:11.241370+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:11.241370+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27867728,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9320,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9220","reported_seq":9326,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:10.544674+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:10.544674+0000","last_peered":"2024-09-11T05:03:10.544674+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:10.544674+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9220,"log_dups_size":0,"ondisk_log_size":9220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620512,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9218,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9146","reported_seq":9264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:11.126009+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:11.126009+0000","last_peered":"2024-09-11T05:03:11.126009+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:11.126009+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9146,"log_dups_size":0,"ondisk_log_size":9146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727584,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9385","reported_seq":9504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:13.877952+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:13.877952+0000","last_peered":"2024-09-11T05:03:13.877952+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:13.877952+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9385,"log_dups_size":0,"ondisk_log_size":9385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27784352,"num_objects":3392,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9383,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":440283624,"num_objects":53751,"num_object_clones":0,"num_object_copies":215004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53751,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132152,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":878678016,"data_stored":878678016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147656,"ondisk_log_size":147656,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721804,"kb_used_data":216700,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93650036,"statfs":{"total":96636764160,"available":95897636864,"internally_reserved":0,"allocated":221900800,"data_stored":221711598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716692,"kb_used_data":216260,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93655148,"statfs":{"total":96636764160,"available":95902871552,"internally_reserved":0,"allocated":221450240,"data_stored":221261038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,5,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631688,"kb_used_data":212708,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93740152,"statfs":{"total":96636764160,"available":95989915648,"internally_reserved":0,"allocated":217812992,"data_stored":217623790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":428994258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,2,9,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715084,"kb_used_data":214204,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93656756,"statfs":{"total":96636764160,"available":95904518144,"internally_reserved":0,"allocated":219344896,"data_stored":219155694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":218656768,"data_stored":218656768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217456640,"data_stored":217456640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220983296,"data_stored":220983296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221581312,"data_stored":221581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:19.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:19.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:19.771 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:19.801 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-11T05:03:17.804666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":435016168,"num_objects":53108,"num_object_clones":0,"num_object_copies":212432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132795,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148299,"ondisk_log_size":148299,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785372,"kb_used_data":851644,"kb_used_omap":14,"kb_used_meta":1933617,"kb_avail":374701988,"statfs":{"total":386547056640,"available":383694835712,"internally_reserved":0,"allocated":872083456,"data_stored":871326648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1980024374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,2,1,9,10,17,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18633888,"num_objects":-2275,"num_object_clones":0,"num_object_copies":-9100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7015,"num_write_kb":-51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2214,"num_bytes_recovered":-15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004967"},"pg_stats":[{"pgid":"1.f","version":"35'9219","reported_seq":9312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.806965+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:15.806965+0000","last_peered":"2024-09-11T05:03:15.806965+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:15.806965+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26992640,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9321","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:13.891868+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:13.891868+0000","last_peered":"2024-09-11T05:03:13.891868+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:13.891868+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27989152,"num_objects":3417,"num_object_clones":0,"num_object_copies":13668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8939","reported_seq":9042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:16.268522+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:16.268522+0000","last_peered":"2024-09-11T05:03:16.268522+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:16.268522+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8939,"log_dups_size":0,"ondisk_log_size":8939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26385000,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8938,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1011}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9154","reported_seq":9261,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:13.721265+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:13.721265+0000","last_peered":"2024-09-11T05:03:13.721265+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:13.721265+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719392,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9205","reported_seq":9327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.681001+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:15.681001+0000","last_peered":"2024-09-11T05:03:15.681001+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:15.681001+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27243680,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9317","reported_seq":9420,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.680536+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:15.680536+0000","last_peered":"2024-09-11T05:03:15.680536+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:15.680536+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9317,"log_dups_size":0,"ondisk_log_size":9317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27222016,"num_objects":3323,"num_object_clones":0,"num_object_copies":13292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9317,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9366","reported_seq":9459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:13.882511+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:13.882511+0000","last_peered":"2024-09-11T05:03:13.882511+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:13.882511+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27554976,"num_objects":3364,"num_object_clones":0,"num_object_copies":13456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1057}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9444","reported_seq":14301,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:15.425442+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:15.425442+0000","last_peered":"2024-09-11T05:03:15.425442+0000","last_clean":"2024-09-11T05:03:15.425442+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:15.425442+0000","last_undegraded":"2024-09-11T05:03:15.425442+0000","last_fullsized":"2024-09-11T05:03:15.425442+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9444,"log_dups_size":0,"ondisk_log_size":9444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27680768,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9443","reported_seq":9563,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.678873+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:15.678873+0000","last_peered":"2024-09-11T05:03:15.678873+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:15.678873+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9443,"log_dups_size":0,"ondisk_log_size":9443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27396416,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9234","reported_seq":9344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:16.374907+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:16.374907+0000","last_peered":"2024-09-11T05:03:16.374907+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:16.374907+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26981536,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9232,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":988}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9301","reported_seq":9417,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.631662+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:15.631662+0000","last_peered":"2024-09-11T05:03:15.631662+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:15.631662+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27455216,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9086","reported_seq":13685,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:15.487092+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:15.487092+0000","last_peered":"2024-09-11T05:03:15.487092+0000","last_clean":"2024-09-11T05:03:15.487092+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:15.487092+0000","last_undegraded":"2024-09-11T05:03:15.487092+0000","last_fullsized":"2024-09-11T05:03:15.487092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9086,"log_dups_size":0,"ondisk_log_size":9086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27017216,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9378","reported_seq":9867,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:16.301183+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:16.301183+0000","last_peered":"2024-09-11T05:03:16.301183+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:16.301183+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9378,"log_dups_size":0,"ondisk_log_size":9378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27400784,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9377,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9294","reported_seq":9400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.752274+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:15.752274+0000","last_peered":"2024-09-11T05:03:15.752274+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:15.752274+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9294,"log_dups_size":0,"ondisk_log_size":9294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27014304,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1034}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9213","reported_seq":9331,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:16.319982+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:16.319982+0000","last_peered":"2024-09-11T05:03:16.319982+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:16.319982+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26178720,"num_objects":3196,"num_object_clones":0,"num_object_copies":12784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9385","reported_seq":9504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:13.877952+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:13.877952+0000","last_peered":"2024-09-11T05:03:13.877952+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:13.877952+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9385,"log_dups_size":0,"ondisk_log_size":9385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27784352,"num_objects":3392,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9383,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":435016168,"num_objects":53108,"num_object_clones":0,"num_object_copies":212432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132795,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":870133760,"data_stored":870133760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148299,"ondisk_log_size":148299,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721780,"kb_used_data":212516,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93650060,"statfs":{"total":96636764160,"available":95897661440,"internally_reserved":0,"allocated":217616384,"data_stored":217427182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716820,"kb_used_data":212216,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93655020,"statfs":{"total":96636764160,"available":95902740480,"internally_reserved":0,"allocated":217309184,"data_stored":217119982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631688,"kb_used_data":212708,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93740152,"statfs":{"total":96636764160,"available":95989915648,"internally_reserved":0,"allocated":217812992,"data_stored":217623790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":428994258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,2,9,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715084,"kb_used_data":214204,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93656756,"statfs":{"total":96636764160,"available":95904518144,"internally_reserved":0,"allocated":219344896,"data_stored":219155694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":218656768,"data_stored":218656768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217456640,"data_stored":217456640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216793088,"data_stored":216793088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217227264,"data_stored":217227264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:19.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:20.449 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:20.472 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-11T05:03:19.805598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":432959976,"num_objects":52857,"num_object_clones":0,"num_object_copies":211428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133046,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148550,"ondisk_log_size":148550,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785524,"kb_used_data":847616,"kb_used_omap":14,"kb_used_meta":1937777,"kb_avail":374701836,"statfs":{"total":386547056640,"available":383694680064,"internally_reserved":0,"allocated":867958784,"data_stored":867201976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1984284214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,7,5,13,13,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18822304,"num_objects":-2298,"num_object_clones":0,"num_object_copies":-9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6992,"num_write_kb":-51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2254,"num_bytes_recovered":-15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005043"},"pg_stats":[{"pgid":"1.f","version":"35'9219","reported_seq":9312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.806965+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:15.806965+0000","last_peered":"2024-09-11T05:03:15.806965+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:15.806965+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26992640,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9388","reported_seq":9502,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.895791+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:18.895791+0000","last_peered":"2024-09-11T05:03:18.895791+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:18.895791+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27440288,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9386,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1092}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8939","reported_seq":9042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:16.268522+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:16.268522+0000","last_peered":"2024-09-11T05:03:16.268522+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:16.268522+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8939,"log_dups_size":0,"ondisk_log_size":8939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26385000,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8938,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1011}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9222","reported_seq":9328,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.996153+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:18.996153+0000","last_peered":"2024-09-11T05:03:18.996153+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:18.996153+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162336,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9205","reported_seq":9327,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.681001+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:15.681001+0000","last_peered":"2024-09-11T05:03:15.681001+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:15.681001+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27243680,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9317","reported_seq":9420,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.680536+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:15.680536+0000","last_peered":"2024-09-11T05:03:15.680536+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:15.680536+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9317,"log_dups_size":0,"ondisk_log_size":9317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27222016,"num_objects":3323,"num_object_clones":0,"num_object_copies":13292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9317,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9418","reported_seq":9511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.832353+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:18.832353+0000","last_peered":"2024-09-11T05:03:18.832353+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:18.832353+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9418,"log_dups_size":0,"ondisk_log_size":9418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27128992,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1005}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9444","reported_seq":14301,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:15.425442+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:15.425442+0000","last_peered":"2024-09-11T05:03:15.425442+0000","last_clean":"2024-09-11T05:03:15.425442+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:15.425442+0000","last_undegraded":"2024-09-11T05:03:15.425442+0000","last_fullsized":"2024-09-11T05:03:15.425442+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9444,"log_dups_size":0,"ondisk_log_size":9444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27680768,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9443","reported_seq":9563,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.678873+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:15.678873+0000","last_peered":"2024-09-11T05:03:15.678873+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:15.678873+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9443,"log_dups_size":0,"ondisk_log_size":9443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27396416,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9234","reported_seq":9344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:16.374907+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:16.374907+0000","last_peered":"2024-09-11T05:03:16.374907+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:16.374907+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26981536,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9232,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":988}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9301","reported_seq":9417,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.631662+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:15.631662+0000","last_peered":"2024-09-11T05:03:15.631662+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:15.631662+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27455216,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9086","reported_seq":13685,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:15.487092+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:15.487092+0000","last_peered":"2024-09-11T05:03:15.487092+0000","last_clean":"2024-09-11T05:03:15.487092+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:15.487092+0000","last_undegraded":"2024-09-11T05:03:15.487092+0000","last_fullsized":"2024-09-11T05:03:15.487092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9086,"log_dups_size":0,"ondisk_log_size":9086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27017216,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9378","reported_seq":9867,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:16.301183+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:16.301183+0000","last_peered":"2024-09-11T05:03:16.301183+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:16.301183+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9378,"log_dups_size":0,"ondisk_log_size":9378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27400784,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9377,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9294","reported_seq":9400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:15.752274+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:15.752274+0000","last_peered":"2024-09-11T05:03:15.752274+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:15.752274+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9294,"log_dups_size":0,"ondisk_log_size":9294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27014304,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1034}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9213","reported_seq":9331,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:16.319982+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:16.319982+0000","last_peered":"2024-09-11T05:03:16.319982+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:16.319982+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26178720,"num_objects":3196,"num_object_clones":0,"num_object_copies":12784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9449","reported_seq":9568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.833280+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:18.833280+0000","last_peered":"2024-09-11T05:03:18.833280+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:18.833280+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27260064,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":432959976,"num_objects":52857,"num_object_clones":0,"num_object_copies":211428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133046,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":865869824,"data_stored":865869824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148550,"ondisk_log_size":148550,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721780,"kb_used_data":212516,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93650060,"statfs":{"total":96636764160,"available":95897661440,"internally_reserved":0,"allocated":217616384,"data_stored":217427182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716820,"kb_used_data":212216,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93655020,"statfs":{"total":96636764160,"available":95902740480,"internally_reserved":0,"allocated":217309184,"data_stored":217119982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631688,"kb_used_data":212708,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93740152,"statfs":{"total":96636764160,"available":95989915648,"internally_reserved":0,"allocated":217812992,"data_stored":217623790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":428994258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,2,9,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715236,"kb_used_data":210176,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93656604,"statfs":{"total":96636764160,"available":95904362496,"internally_reserved":0,"allocated":215220224,"data_stored":215031022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,0,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214392832,"data_stored":214392832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217456640,"data_stored":217456640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216793088,"data_stored":216793088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217227264,"data_stored":217227264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:20.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:20.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9219", 'reported_seq': 9312, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:15.806965+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:15.806965+0000', 'last_peered': '2024-09-11T05:03:15.806965+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:15.806965+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9219, 'log_dups_size': 0, 'ondisk_log_size': 9219, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26992640, 'num_objects': 3295, 'num_object_clones': 0, 'num_object_copies': 13180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9219, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:20.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:20.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9388", 'reported_seq': 9502, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:18.895791+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:18.895791+0000', 'last_peered': '2024-09-11T05:03:18.895791+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:18.895791+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9388, 'log_dups_size': 0, 'ondisk_log_size': 9388, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27440288, 'num_objects': 3350, 'num_object_clones': 0, 'num_object_copies': 13400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9386, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1092}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:20.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:20.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8939", 'reported_seq': 9042, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:16.268522+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:16.268522+0000', 'last_peered': '2024-09-11T05:03:16.268522+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:16.268522+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8939, 'log_dups_size': 0, 'ondisk_log_size': 8939, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26385000, 'num_objects': 3222, 'num_object_clones': 0, 'num_object_copies': 12888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3222, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8938, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1011}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:20.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9222", 'reported_seq': 9328, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:18.996153+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:18.996153+0000', 'last_peered': '2024-09-11T05:03:18.996153+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:18.996153+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9222, 'log_dups_size': 0, 'ondisk_log_size': 9222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26162336, 'num_objects': 3194, 'num_object_clones': 0, 'num_object_copies': 12776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9220, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 933}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9205", 'reported_seq': 9327, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:15.681001+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:15.681001+0000', 'last_peered': '2024-09-11T05:03:15.681001+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:15.681001+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9205, 'log_dups_size': 0, 'ondisk_log_size': 9205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27243680, 'num_objects': 3326, 'num_object_clones': 0, 'num_object_copies': 13304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9203, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1105}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9317", 'reported_seq': 9420, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:15.680536+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:15.680536+0000', 'last_peered': '2024-09-11T05:03:15.680536+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:15.680536+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9317, 'log_dups_size': 0, 'ondisk_log_size': 9317, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27222016, 'num_objects': 3323, 'num_object_clones': 0, 'num_object_copies': 13292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9317, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1029}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9418", 'reported_seq': 9511, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:18.832353+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:18.832353+0000', 'last_peered': '2024-09-11T05:03:18.832353+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:18.832353+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9418, 'log_dups_size': 0, 'ondisk_log_size': 9418, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27128992, 'num_objects': 3312, 'num_object_clones': 0, 'num_object_copies': 13248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9416, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1005}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9443", 'reported_seq': 9563, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:15.678873+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:15.678873+0000', 'last_peered': '2024-09-11T05:03:15.678873+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:15.678873+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9443, 'log_dups_size': 0, 'ondisk_log_size': 9443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27396416, 'num_objects': 3345, 'num_object_clones': 0, 'num_object_copies': 13380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9440, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 979}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:20.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9234", 'reported_seq': 9344, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:16.374907+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:16.374907+0000', 'last_peered': '2024-09-11T05:03:16.374907+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:16.374907+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9234, 'log_dups_size': 0, 'ondisk_log_size': 9234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26981536, 'num_objects': 3294, 'num_object_clones': 0, 'num_object_copies': 13176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9232, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 988}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9301", 'reported_seq': 9417, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:15.631662+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:15.631662+0000', 'last_peered': '2024-09-11T05:03:15.631662+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:15.631662+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9301, 'log_dups_size': 0, 'ondisk_log_size': 9301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27455216, 'num_objects': 3352, 'num_object_clones': 0, 'num_object_copies': 13408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9298, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1054}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9378", 'reported_seq': 9867, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:16.301183+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:16.301183+0000', 'last_peered': '2024-09-11T05:03:16.301183+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:16.301183+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9378, 'log_dups_size': 0, 'ondisk_log_size': 9378, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27400784, 'num_objects': 3345, 'num_object_clones': 0, 'num_object_copies': 13380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2131, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9377, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 203, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1214}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9294", 'reported_seq': 9400, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:15.752274+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:15.752274+0000', 'last_peered': '2024-09-11T05:03:15.752274+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:15.752274+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9294, 'log_dups_size': 0, 'ondisk_log_size': 9294, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27014304, 'num_objects': 3298, 'num_object_clones': 0, 'num_object_copies': 13192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9292, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1034}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9213", 'reported_seq': 9331, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:16.319982+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:16.319982+0000', 'last_peered': '2024-09-11T05:03:16.319982+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:16.319982+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9213, 'log_dups_size': 0, 'ondisk_log_size': 9213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26178720, 'num_objects': 3196, 'num_object_clones': 0, 'num_object_copies': 12784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9211, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:20.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9449", 'reported_seq': 9568, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:18.833280+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:18.833280+0000', 'last_peered': '2024-09-11T05:03:18.833280+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:18.833280+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9449, 'log_dups_size': 0, 'ondisk_log_size': 9449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27260064, 'num_objects': 3328, 'num_object_clones': 0, 'num_object_copies': 13312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9447, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1040}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:20.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:21.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:21.263 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31822/211428 objects degraded (15.051%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":528,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52857,"data_bytes":432959976,"bytes_used":2852376576,"bytes_avail":383694680064,"bytes_total":386547056640,"degraded_objects":31822,"degraded_total":211428,"degraded_ratio":0.15050986624288173},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:21.286 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:03:21.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:21.887 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-11T05:03:21.806442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":426308072,"num_objects":52045,"num_object_clones":0,"num_object_copies":208180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133858,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149362,"ondisk_log_size":149362,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2756240,"kb_used_data":835004,"kb_used_omap":14,"kb_used_meta":1921137,"kb_avail":374731120,"statfs":{"total":386547056640,"available":383724666880,"internally_reserved":0,"allocated":855044096,"data_stored":854287288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1967244854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,11,8,15,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23360672,"num_objects":-2852,"num_object_clones":0,"num_object_copies":-11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6438,"num_write_kb":-51292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2212,"num_bytes_recovered":-15564800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004968"},"pg_stats":[{"pgid":"1.f","version":"35'9287","reported_seq":9381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.693096+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:20.693096+0000","last_peered":"2024-09-11T05:03:20.693096+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:20.693096+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26435584,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9388","reported_seq":9502,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.895791+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:18.895791+0000","last_peered":"2024-09-11T05:03:18.895791+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:18.895791+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27440288,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9386,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1092}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8992","reported_seq":9096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.082706+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:21.082706+0000","last_peered":"2024-09-11T05:03:21.082706+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:21.082706+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25950824,"num_objects":3169,"num_object_clones":0,"num_object_copies":12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8991,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":958}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9222","reported_seq":9328,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.996153+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:18.996153+0000","last_peered":"2024-09-11T05:03:18.996153+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:18.996153+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162336,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9278","reported_seq":9400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.827166+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:20.827166+0000","last_peered":"2024-09-11T05:03:20.827166+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:20.827166+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26645664,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9375","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.806060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:20.806060+0000","last_peered":"2024-09-11T05:03:20.806060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:20.806060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26746880,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9418","reported_seq":9511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.832353+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:18.832353+0000","last_peered":"2024-09-11T05:03:18.832353+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:18.832353+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9418,"log_dups_size":0,"ondisk_log_size":9418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27128992,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1005}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9517","reported_seq":14374,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:20.488236+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:20.488236+0000","last_peered":"2024-09-11T05:03:20.488236+0000","last_clean":"2024-09-11T05:03:20.488236+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:20.488236+0000","last_undegraded":"2024-09-11T05:03:20.488236+0000","last_fullsized":"2024-09-11T05:03:20.488236+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27082752,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9515","reported_seq":9637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.744150+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:20.744150+0000","last_peered":"2024-09-11T05:03:20.744150+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:20.744150+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26806592,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9312","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.377077+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:21.377077+0000","last_peered":"2024-09-11T05:03:21.377077+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:21.377077+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9364","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.726794+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:20.726794+0000","last_peered":"2024-09-11T05:03:20.726794+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:20.726794+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26939120,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9156","reported_seq":13756,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:20.487940+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:20.487940+0000","last_peered":"2024-09-11T05:03:20.487940+0000","last_clean":"2024-09-11T05:03:20.487940+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:20.487940+0000","last_undegraded":"2024-09-11T05:03:20.487940+0000","last_fullsized":"2024-09-11T05:03:20.487940+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26443776,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2966,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9454","reported_seq":10026,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.368839+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:21.368839+0000","last_peered":"2024-09-11T05:03:21.368839+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:21.368839+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26778192,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9453,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9352","reported_seq":9458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.703418+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:20.703418+0000","last_peered":"2024-09-11T05:03:20.703418+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:20.703418+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539168,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9283","reported_seq":9401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.275392+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:21.275392+0000","last_peered":"2024-09-11T05:03:21.275392+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:21.275392+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605280,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9449","reported_seq":9568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.833280+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:18.833280+0000","last_peered":"2024-09-11T05:03:18.833280+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:18.833280+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27260064,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":426308072,"num_objects":52045,"num_object_clones":0,"num_object_copies":208180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133858,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":853028864,"data_stored":853028864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149362,"ondisk_log_size":149362,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705140,"kb_used_data":208356,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93666700,"statfs":{"total":96636764160,"available":95914700800,"internally_reserved":0,"allocated":213356544,"data_stored":213167342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,4,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716752,"kb_used_data":208000,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93655088,"statfs":{"total":96636764160,"available":95902810112,"internally_reserved":0,"allocated":212992000,"data_stored":212802798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619112,"kb_used_data":208472,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93752728,"statfs":{"total":96636764160,"available":96002793472,"internally_reserved":0,"allocated":213475328,"data_stored":213286126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420474578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715236,"kb_used_data":210176,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93656604,"statfs":{"total":96636764160,"available":95904362496,"internally_reserved":0,"allocated":215220224,"data_stored":215031022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,0,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214392832,"data_stored":214392832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213180416,"data_stored":213180416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212529152,"data_stored":212529152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212926464,"data_stored":212926464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:24.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:25.561 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:25.562 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:25.583 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-09-11T05:03:23.807321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":426308072,"num_objects":52045,"num_object_clones":0,"num_object_copies":208180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133858,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149362,"ondisk_log_size":149362,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2756240,"kb_used_data":835004,"kb_used_omap":14,"kb_used_meta":1921137,"kb_avail":374731120,"statfs":{"total":386547056640,"available":383724666880,"internally_reserved":0,"allocated":855044096,"data_stored":854287288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1967244854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,11,8,15,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17375232,"num_objects":-2121,"num_object_clones":0,"num_object_copies":-8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2121,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005112"},"pg_stats":[{"pgid":"1.f","version":"35'9287","reported_seq":9381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.693096+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:20.693096+0000","last_peered":"2024-09-11T05:03:20.693096+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:20.693096+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26435584,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9388","reported_seq":9502,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.895791+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:18.895791+0000","last_peered":"2024-09-11T05:03:18.895791+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:18.895791+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27440288,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9386,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1092}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8992","reported_seq":9096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.082706+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:21.082706+0000","last_peered":"2024-09-11T05:03:21.082706+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:21.082706+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25950824,"num_objects":3169,"num_object_clones":0,"num_object_copies":12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8991,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":958}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9222","reported_seq":9328,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.996153+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:18.996153+0000","last_peered":"2024-09-11T05:03:18.996153+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:18.996153+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162336,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9278","reported_seq":9400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.827166+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:20.827166+0000","last_peered":"2024-09-11T05:03:20.827166+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:20.827166+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26645664,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9375","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.806060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:20.806060+0000","last_peered":"2024-09-11T05:03:20.806060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:20.806060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26746880,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9418","reported_seq":9511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.832353+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:18.832353+0000","last_peered":"2024-09-11T05:03:18.832353+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:18.832353+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9418,"log_dups_size":0,"ondisk_log_size":9418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27128992,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1005}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9517","reported_seq":14374,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:20.488236+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:20.488236+0000","last_peered":"2024-09-11T05:03:20.488236+0000","last_clean":"2024-09-11T05:03:20.488236+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:20.488236+0000","last_undegraded":"2024-09-11T05:03:20.488236+0000","last_fullsized":"2024-09-11T05:03:20.488236+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27082752,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9515","reported_seq":9637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.744150+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:20.744150+0000","last_peered":"2024-09-11T05:03:20.744150+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:20.744150+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26806592,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9312","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.377077+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:21.377077+0000","last_peered":"2024-09-11T05:03:21.377077+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:21.377077+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9364","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.726794+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:20.726794+0000","last_peered":"2024-09-11T05:03:20.726794+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:20.726794+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26939120,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9156","reported_seq":13756,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:20.487940+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:20.487940+0000","last_peered":"2024-09-11T05:03:20.487940+0000","last_clean":"2024-09-11T05:03:20.487940+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:20.487940+0000","last_undegraded":"2024-09-11T05:03:20.487940+0000","last_fullsized":"2024-09-11T05:03:20.487940+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26443776,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2966,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9454","reported_seq":10026,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.368839+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:21.368839+0000","last_peered":"2024-09-11T05:03:21.368839+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:21.368839+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26778192,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9453,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9352","reported_seq":9458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.703418+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:20.703418+0000","last_peered":"2024-09-11T05:03:20.703418+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:20.703418+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539168,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9283","reported_seq":9401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.275392+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:21.275392+0000","last_peered":"2024-09-11T05:03:21.275392+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:21.275392+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605280,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9449","reported_seq":9568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:18.833280+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:18.833280+0000","last_peered":"2024-09-11T05:03:18.833280+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:18.833280+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27260064,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":426308072,"num_objects":52045,"num_object_clones":0,"num_object_copies":208180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133858,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":853028864,"data_stored":853028864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149362,"ondisk_log_size":149362,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705140,"kb_used_data":208356,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93666700,"statfs":{"total":96636764160,"available":95914700800,"internally_reserved":0,"allocated":213356544,"data_stored":213167342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,4,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716752,"kb_used_data":208000,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93655088,"statfs":{"total":96636764160,"available":95902810112,"internally_reserved":0,"allocated":212992000,"data_stored":212802798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619112,"kb_used_data":208472,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93752728,"statfs":{"total":96636764160,"available":96002793472,"internally_reserved":0,"allocated":213475328,"data_stored":213286126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420474578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715236,"kb_used_data":210176,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93656604,"statfs":{"total":96636764160,"available":95904362496,"internally_reserved":0,"allocated":215220224,"data_stored":215031022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,0,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214392832,"data_stored":214392832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213180416,"data_stored":213180416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212529152,"data_stored":212529152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212926464,"data_stored":212926464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:25.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:26.270 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:26.270 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:26.298 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-11T05:03:25.807948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424366568,"num_objects":51808,"num_object_clones":0,"num_object_copies":207232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134095,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149599,"ondisk_log_size":149599,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2756324,"kb_used_data":830948,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374731036,"statfs":{"total":386547056640,"available":383724580864,"internally_reserved":0,"allocated":850890752,"data_stored":850133944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1971504694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,5,10,10,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19316736,"num_objects":-2358,"num_object_clones":0,"num_object_copies":-9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005051"},"pg_stats":[{"pgid":"1.f","version":"35'9287","reported_seq":9381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.693096+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:20.693096+0000","last_peered":"2024-09-11T05:03:20.693096+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:20.693096+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26435584,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9461","reported_seq":9575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:24.183616+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:24.183616+0000","last_peered":"2024-09-11T05:03:24.183616+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:24.183616+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9461,"log_dups_size":0,"ondisk_log_size":9461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26842272,"num_objects":3277,"num_object_clones":0,"num_object_copies":13108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9459,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8992","reported_seq":9096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.082706+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:21.082706+0000","last_peered":"2024-09-11T05:03:21.082706+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:21.082706+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25950824,"num_objects":3169,"num_object_clones":0,"num_object_copies":12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8991,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":958}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9268","reported_seq":9375,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:24.078669+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:24.078669+0000","last_peered":"2024-09-11T05:03:24.078669+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:24.078669+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9268,"log_dups_size":0,"ondisk_log_size":9268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25785504,"num_objects":3148,"num_object_clones":0,"num_object_copies":12592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9266,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9278","reported_seq":9400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.827166+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:20.827166+0000","last_peered":"2024-09-11T05:03:20.827166+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:20.827166+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26645664,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9375","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.806060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:20.806060+0000","last_peered":"2024-09-11T05:03:20.806060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:20.806060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26746880,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9480","reported_seq":9572,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:24.181644+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:24.181644+0000","last_peered":"2024-09-11T05:03:24.181644+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:24.181644+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9480,"log_dups_size":0,"ondisk_log_size":9480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26621088,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9478,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":943}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9517","reported_seq":14374,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:20.488236+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:20.488236+0000","last_peered":"2024-09-11T05:03:20.488236+0000","last_clean":"2024-09-11T05:03:20.488236+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:20.488236+0000","last_undegraded":"2024-09-11T05:03:20.488236+0000","last_fullsized":"2024-09-11T05:03:20.488236+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27082752,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9515","reported_seq":9637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.744150+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:20.744150+0000","last_peered":"2024-09-11T05:03:20.744150+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:20.744150+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26806592,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9312","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.377077+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:21.377077+0000","last_peered":"2024-09-11T05:03:21.377077+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:21.377077+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9364","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.726794+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:20.726794+0000","last_peered":"2024-09-11T05:03:20.726794+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:20.726794+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26939120,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9156","reported_seq":13756,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:20.487940+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:20.487940+0000","last_peered":"2024-09-11T05:03:20.487940+0000","last_clean":"2024-09-11T05:03:20.487940+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:20.487940+0000","last_undegraded":"2024-09-11T05:03:20.487940+0000","last_fullsized":"2024-09-11T05:03:20.487940+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26443776,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2966,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9454","reported_seq":10026,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.368839+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:21.368839+0000","last_peered":"2024-09-11T05:03:21.368839+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:21.368839+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26778192,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9453,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9352","reported_seq":9458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.703418+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:20.703418+0000","last_peered":"2024-09-11T05:03:20.703418+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:20.703418+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539168,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9283","reported_seq":9401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.275392+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:21.275392+0000","last_peered":"2024-09-11T05:03:21.275392+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:21.275392+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605280,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9505","reported_seq":9624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:24.019257+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:24.019257+0000","last_peered":"2024-09-11T05:03:24.019257+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:24.019257+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9505,"log_dups_size":0,"ondisk_log_size":9505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26801312,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9503,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":424366568,"num_objects":51808,"num_object_clones":0,"num_object_copies":207232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134095,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":848764928,"data_stored":848764928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149599,"ondisk_log_size":149599,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705140,"kb_used_data":208356,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93666700,"statfs":{"total":96636764160,"available":95914700800,"internally_reserved":0,"allocated":213356544,"data_stored":213167342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,4,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716752,"kb_used_data":208000,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93655088,"statfs":{"total":96636764160,"available":95902810112,"internally_reserved":0,"allocated":212992000,"data_stored":212802798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619112,"kb_used_data":208472,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93752728,"statfs":{"total":96636764160,"available":96002793472,"internally_reserved":0,"allocated":213475328,"data_stored":213286126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420474578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715320,"kb_used_data":206120,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93656520,"statfs":{"total":96636764160,"available":95904276480,"internally_reserved":0,"allocated":211066880,"data_stored":210877678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210128896,"data_stored":210128896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213180416,"data_stored":213180416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212529152,"data_stored":212529152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212926464,"data_stored":212926464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:26.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:26.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9287", 'reported_seq': 9381, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:20.693096+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:20.693096+0000', 'last_peered': '2024-09-11T05:03:20.693096+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:20.693096+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9287, 'log_dups_size': 0, 'ondisk_log_size': 9287, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26435584, 'num_objects': 3227, 'num_object_clones': 0, 'num_object_copies': 12908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9287, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 936}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:26.299 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:26.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9461", 'reported_seq': 9575, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:24.183616+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:24.183616+0000', 'last_peered': '2024-09-11T05:03:24.183616+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:24.183616+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9461, 'log_dups_size': 0, 'ondisk_log_size': 9461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26842272, 'num_objects': 3277, 'num_object_clones': 0, 'num_object_copies': 13108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9459, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8992", 'reported_seq': 9096, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:21.082706+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:21.082706+0000', 'last_peered': '2024-09-11T05:03:21.082706+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:21.082706+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 8992, 'log_dups_size': 0, 'ondisk_log_size': 8992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25950824, 'num_objects': 3169, 'num_object_clones': 0, 'num_object_copies': 12676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3169, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8991, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 958}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9268", 'reported_seq': 9375, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:24.078669+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:24.078669+0000', 'last_peered': '2024-09-11T05:03:24.078669+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:24.078669+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9268, 'log_dups_size': 0, 'ondisk_log_size': 9268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25785504, 'num_objects': 3148, 'num_object_clones': 0, 'num_object_copies': 12592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9266, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9278", 'reported_seq': 9400, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:20.827166+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:20.827166+0000', 'last_peered': '2024-09-11T05:03:20.827166+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:20.827166+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9278, 'log_dups_size': 0, 'ondisk_log_size': 9278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26645664, 'num_objects': 3253, 'num_object_clones': 0, 'num_object_copies': 13012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9276, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1032}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9375", 'reported_seq': 9480, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:20.806060+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:20.806060+0000', 'last_peered': '2024-09-11T05:03:20.806060+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:20.806060+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9375, 'log_dups_size': 0, 'ondisk_log_size': 9375, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26746880, 'num_objects': 3265, 'num_object_clones': 0, 'num_object_copies': 13060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9375, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 971}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:26.300 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9480", 'reported_seq': 9572, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:24.181644+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:24.181644+0000', 'last_peered': '2024-09-11T05:03:24.181644+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:24.181644+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9480, 'log_dups_size': 0, 'ondisk_log_size': 9480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26621088, 'num_objects': 3250, 'num_object_clones': 0, 'num_object_copies': 13000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9478, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 943}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9515", 'reported_seq': 9637, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:20.744150+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:20.744150+0000', 'last_peered': '2024-09-11T05:03:20.744150+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:20.744150+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9515, 'log_dups_size': 0, 'ondisk_log_size': 9515, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26806592, 'num_objects': 3273, 'num_object_clones': 0, 'num_object_copies': 13092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9512, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 907}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9312", 'reported_seq': 9424, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:21.377077+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:21.377077+0000', 'last_peered': '2024-09-11T05:03:21.377077+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:21.377077+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9312, 'log_dups_size': 0, 'ondisk_log_size': 9312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26342560, 'num_objects': 3216, 'num_object_clones': 0, 'num_object_copies': 12864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9310, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 910}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9364", 'reported_seq': 9480, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:20.726794+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:20.726794+0000', 'last_peered': '2024-09-11T05:03:20.726794+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:20.726794+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9364, 'log_dups_size': 0, 'ondisk_log_size': 9364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26939120, 'num_objects': 3289, 'num_object_clones': 0, 'num_object_copies': 13156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9361, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 991}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9454", 'reported_seq': 10026, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:21.368839+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:21.368839+0000', 'last_peered': '2024-09-11T05:03:21.368839+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:21.368839+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9454, 'log_dups_size': 0, 'ondisk_log_size': 9454, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26778192, 'num_objects': 3269, 'num_object_clones': 0, 'num_object_copies': 13076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2089, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9453, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 245, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9352", 'reported_seq': 9458, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:20.703418+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:20.703418+0000', 'last_peered': '2024-09-11T05:03:20.703418+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:20.703418+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9352, 'log_dups_size': 0, 'ondisk_log_size': 9352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26539168, 'num_objects': 3240, 'num_object_clones': 0, 'num_object_copies': 12960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9350, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:26.301 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:26.302 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9283", 'reported_seq': 9401, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:21.275392+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:21.275392+0000', 'last_peered': '2024-09-11T05:03:21.275392+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:21.275392+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9283, 'log_dups_size': 0, 'ondisk_log_size': 9283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25605280, 'num_objects': 3126, 'num_object_clones': 0, 'num_object_copies': 12504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9281, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 823}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:26.302 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:26.302 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9505", 'reported_seq': 9624, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:24.019257+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:24.019257+0000', 'last_peered': '2024-09-11T05:03:24.019257+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:24.019257+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9505, 'log_dups_size': 0, 'ondisk_log_size': 9505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26801312, 'num_objects': 3272, 'num_object_clones': 0, 'num_object_copies': 13088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9503, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 984}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:26.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:27.098 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31780/207232 objects degraded (15.335%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":534,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51808,"data_bytes":424366568,"bytes_used":2822475776,"bytes_avail":383724580864,"bytes_total":386547056640,"degraded_objects":31780,"degraded_total":207232,"degraded_ratio":0.15335469425571341,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:03:27.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:27.768 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:27.787 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-11T05:03:25.807948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424366568,"num_objects":51808,"num_object_clones":0,"num_object_copies":207232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134095,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149599,"ondisk_log_size":149599,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2756324,"kb_used_data":830948,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374731036,"statfs":{"total":386547056640,"available":383724580864,"internally_reserved":0,"allocated":850890752,"data_stored":850133944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1971504694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,5,10,10,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19316736,"num_objects":-2358,"num_object_clones":0,"num_object_copies":-9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005051"},"pg_stats":[{"pgid":"1.f","version":"35'9287","reported_seq":9381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.693096+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:20.693096+0000","last_peered":"2024-09-11T05:03:20.693096+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:20.693096+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26435584,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9461","reported_seq":9575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:24.183616+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:24.183616+0000","last_peered":"2024-09-11T05:03:24.183616+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:24.183616+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9461,"log_dups_size":0,"ondisk_log_size":9461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26842272,"num_objects":3277,"num_object_clones":0,"num_object_copies":13108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9459,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8992","reported_seq":9096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.082706+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:21.082706+0000","last_peered":"2024-09-11T05:03:21.082706+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:21.082706+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25950824,"num_objects":3169,"num_object_clones":0,"num_object_copies":12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8991,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":958}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9268","reported_seq":9375,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:24.078669+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:24.078669+0000","last_peered":"2024-09-11T05:03:24.078669+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:24.078669+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9268,"log_dups_size":0,"ondisk_log_size":9268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25785504,"num_objects":3148,"num_object_clones":0,"num_object_copies":12592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9266,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9278","reported_seq":9400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.827166+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:20.827166+0000","last_peered":"2024-09-11T05:03:20.827166+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:20.827166+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26645664,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9375","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.806060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:20.806060+0000","last_peered":"2024-09-11T05:03:20.806060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:20.806060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26746880,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9480","reported_seq":9572,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:24.181644+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:24.181644+0000","last_peered":"2024-09-11T05:03:24.181644+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:24.181644+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9480,"log_dups_size":0,"ondisk_log_size":9480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26621088,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9478,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":943}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9517","reported_seq":14374,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:20.488236+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:20.488236+0000","last_peered":"2024-09-11T05:03:20.488236+0000","last_clean":"2024-09-11T05:03:20.488236+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:20.488236+0000","last_undegraded":"2024-09-11T05:03:20.488236+0000","last_fullsized":"2024-09-11T05:03:20.488236+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27082752,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9515","reported_seq":9637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.744150+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:20.744150+0000","last_peered":"2024-09-11T05:03:20.744150+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:20.744150+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26806592,"num_objects":3273,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9312","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.377077+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:21.377077+0000","last_peered":"2024-09-11T05:03:21.377077+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:21.377077+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9312,"log_dups_size":0,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342560,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9364","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.726794+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:20.726794+0000","last_peered":"2024-09-11T05:03:20.726794+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:20.726794+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26939120,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9156","reported_seq":13756,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:20.487940+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:20.487940+0000","last_peered":"2024-09-11T05:03:20.487940+0000","last_clean":"2024-09-11T05:03:20.487940+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:20.487940+0000","last_undegraded":"2024-09-11T05:03:20.487940+0000","last_fullsized":"2024-09-11T05:03:20.487940+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26443776,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2966,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9454","reported_seq":10026,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.368839+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:21.368839+0000","last_peered":"2024-09-11T05:03:21.368839+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:21.368839+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26778192,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9453,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9352","reported_seq":9458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:20.703418+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:20.703418+0000","last_peered":"2024-09-11T05:03:20.703418+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:20.703418+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539168,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":976}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9283","reported_seq":9401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:21.275392+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:21.275392+0000","last_peered":"2024-09-11T05:03:21.275392+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:21.275392+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9283,"log_dups_size":0,"ondisk_log_size":9283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605280,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9505","reported_seq":9624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:24.019257+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:24.019257+0000","last_peered":"2024-09-11T05:03:24.019257+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:24.019257+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9505,"log_dups_size":0,"ondisk_log_size":9505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26801312,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9503,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":424366568,"num_objects":51808,"num_object_clones":0,"num_object_copies":207232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134095,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":848764928,"data_stored":848764928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149599,"ondisk_log_size":149599,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705140,"kb_used_data":208356,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93666700,"statfs":{"total":96636764160,"available":95914700800,"internally_reserved":0,"allocated":213356544,"data_stored":213167342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,4,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716752,"kb_used_data":208000,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93655088,"statfs":{"total":96636764160,"available":95902810112,"internally_reserved":0,"allocated":212992000,"data_stored":212802798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619112,"kb_used_data":208472,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93752728,"statfs":{"total":96636764160,"available":96002793472,"internally_reserved":0,"allocated":213475328,"data_stored":213286126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":420474578},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715320,"kb_used_data":206120,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93656520,"statfs":{"total":96636764160,"available":95904276480,"internally_reserved":0,"allocated":211066880,"data_stored":210877678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210128896,"data_stored":210128896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213180416,"data_stored":213180416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212529152,"data_stored":212529152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212926464,"data_stored":212926464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:30.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:31.440 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:31.441 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:31.469 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-09-11T05:03:29.809619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416035304,"num_objects":50791,"num_object_clones":0,"num_object_copies":203164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135112,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150616,"ondisk_log_size":150616,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2723192,"kb_used_data":814456,"kb_used_omap":14,"kb_used_meta":1908657,"kb_avail":374764168,"statfs":{"total":386547056640,"available":383758508032,"internally_reserved":0,"allocated":834002944,"data_stored":833246136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1954465334},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,1,7,3,11,18,23,17,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18980864,"num_objects":-2317,"num_object_clones":0,"num_object_copies":-9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2317,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004952"},"pg_stats":[{"pgid":"1.f","version":"35'9353","reported_seq":9446,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:25.867714+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:25.867714+0000","last_peered":"2024-09-11T05:03:25.867714+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:25.867714+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9353,"log_dups_size":0,"ondisk_log_size":9353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25894912,"num_objects":3161,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9512","reported_seq":9626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.089402+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:29.089402+0000","last_peered":"2024-09-11T05:03:29.089402+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:29.089402+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424480,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9051","reported_seq":9155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:26.156808+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:26.156808+0000","last_peered":"2024-09-11T05:03:26.156808+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:26.156808+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9051,"log_dups_size":0,"ondisk_log_size":9051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25467496,"num_objects":3110,"num_object_clones":0,"num_object_copies":12440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9050,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":899}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9319","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.208926+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:29.208926+0000","last_peered":"2024-09-11T05:03:29.208926+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:29.208926+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9319,"log_dups_size":0,"ondisk_log_size":9319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25367712,"num_objects":3097,"num_object_clones":0,"num_object_copies":12388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9317,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9358","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:25.787796+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:25.787796+0000","last_peered":"2024-09-11T05:03:25.787796+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:25.787796+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9358,"log_dups_size":0,"ondisk_log_size":9358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25990304,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9356,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":952}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9439","reported_seq":9544,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:25.670885+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:25.670885+0000","last_peered":"2024-09-11T05:03:25.670885+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:25.670885+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9439,"log_dups_size":0,"ondisk_log_size":9439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26222592,"num_objects":3201,"num_object_clones":0,"num_object_copies":12804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9439,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9544","reported_seq":9637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.283917+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:29.283917+0000","last_peered":"2024-09-11T05:03:29.283917+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:29.283917+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096800,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":879}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9579","reported_seq":14436,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:25.589578+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:25.589578+0000","last_peered":"2024-09-11T05:03:25.589578+0000","last_clean":"2024-09-11T05:03:25.589578+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:25.589578+0000","last_undegraded":"2024-09-11T05:03:25.589578+0000","last_fullsized":"2024-09-11T05:03:25.589578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9579,"log_dups_size":0,"ondisk_log_size":9579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26574848,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9586","reported_seq":9708,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:25.896353+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:25.896353+0000","last_peered":"2024-09-11T05:03:25.896353+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:25.896353+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9586,"log_dups_size":0,"ondisk_log_size":9586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26224960,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":836}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9369","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:26.079466+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:26.079466+0000","last_peered":"2024-09-11T05:03:26.079466+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:26.079466+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25875616,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":853}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9425","reported_seq":9541,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:25.932704+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:25.932704+0000","last_peered":"2024-09-11T05:03:25.932704+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:25.932704+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26439408,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9422,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9233","reported_seq":13831,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:25.796207+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:25.796207+0000","last_peered":"2024-09-11T05:03:25.796207+0000","last_clean":"2024-09-11T05:03:25.796207+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:25.796207+0000","last_undegraded":"2024-09-11T05:03:25.796207+0000","last_fullsized":"2024-09-11T05:03:25.796207+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9233,"log_dups_size":0,"ondisk_log_size":9233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25812992,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9514","reported_seq":10191,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:26.322683+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:26.322683+0000","last_peered":"2024-09-11T05:03:26.322683+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:26.322683+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9514,"log_dups_size":0,"ondisk_log_size":9514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26286672,"num_objects":3209,"num_object_clones":0,"num_object_copies":12836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9513,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1172}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9420","reported_seq":9526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:25.838229+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:25.838229+0000","last_peered":"2024-09-11T05:03:25.838229+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:25.838229+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9420,"log_dups_size":0,"ondisk_log_size":9420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25982112,"num_objects":3172,"num_object_clones":0,"num_object_copies":12688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":908}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9340","reported_seq":9458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:26.333698+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:26.333698+0000","last_peered":"2024-09-11T05:03:26.333698+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:26.333698+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138336,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9338,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":766}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9574","reported_seq":9693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.203550+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:29.203550+0000","last_peered":"2024-09-11T05:03:29.203550+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:29.203550+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26236064,"num_objects":3203,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":416035304,"num_objects":50791,"num_object_clones":0,"num_object_copies":203164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135112,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":831811584,"data_stored":831811584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150616,"ondisk_log_size":150616,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705140,"kb_used_data":204196,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93666700,"statfs":{"total":96636764160,"available":95914700800,"internally_reserved":0,"allocated":209096704,"data_stored":208907502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,0,1,9,3,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700108,"kb_used_data":203836,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93671732,"statfs":{"total":96636764160,"available":95919853568,"internally_reserved":0,"allocated":208728064,"data_stored":208538862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,0,0,1,2,3,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619240,"kb_used_data":204440,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93752600,"statfs":{"total":96636764160,"available":96002662400,"internally_reserved":0,"allocated":209346560,"data_stored":209157358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":424734418},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698704,"kb_used_data":201984,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93673136,"statfs":{"total":96636764160,"available":95921291264,"internally_reserved":0,"allocated":206831616,"data_stored":206642414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,7,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205938688,"data_stored":205938688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208871424,"data_stored":208871424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":208338944,"data_stored":208338944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":208662528,"data_stored":208662528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:31.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:32.055 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:32.055 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:32.076 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-11T05:03:31.810481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409506280,"num_objects":49994,"num_object_clones":0,"num_object_copies":199976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135909,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151413,"ondisk_log_size":151413,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741260,"kb_used_data":801976,"kb_used_omap":14,"kb_used_meta":1939185,"kb_avail":374746100,"statfs":{"total":386547056640,"available":383740006400,"internally_reserved":0,"allocated":821223424,"data_stored":820466616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1985726006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,22,10,17,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23453696,"num_objects":-2863,"num_object_clones":0,"num_object_copies":-11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004882"},"pg_stats":[{"pgid":"1.f","version":"35'9412","reported_seq":9505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.008778+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:31.008778+0000","last_peered":"2024-09-11T05:03:31.008778+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:31.008778+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25411584,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9512","reported_seq":9626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.089402+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:29.089402+0000","last_peered":"2024-09-11T05:03:29.089402+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:29.089402+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424480,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9114","reported_seq":9217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.386920+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:31.386920+0000","last_peered":"2024-09-11T05:03:31.386920+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:31.386920+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9114,"log_dups_size":0,"ondisk_log_size":9114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24951400,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9113,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":836}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9319","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.208926+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:29.208926+0000","last_peered":"2024-09-11T05:03:29.208926+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:29.208926+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9319,"log_dups_size":0,"ondisk_log_size":9319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25367712,"num_objects":3097,"num_object_clones":0,"num_object_copies":12388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9317,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9429","reported_seq":9551,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.033572+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:31.033572+0000","last_peered":"2024-09-11T05:03:31.033572+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:31.033572+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25408672,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9427,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9493","reported_seq":9598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.754918+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:30.754918+0000","last_peered":"2024-09-11T05:03:30.754918+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:30.754918+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25780224,"num_objects":3147,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9544","reported_seq":9637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.283917+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:29.283917+0000","last_peered":"2024-09-11T05:03:29.283917+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:29.283917+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096800,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":879}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9657","reported_seq":14514,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:30.473813+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:30.473813+0000","last_peered":"2024-09-11T05:03:30.473813+0000","last_clean":"2024-09-11T05:03:30.473813+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:30.473813+0000","last_undegraded":"2024-09-11T05:03:30.473813+0000","last_fullsized":"2024-09-11T05:03:30.473813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9657,"log_dups_size":0,"ondisk_log_size":9657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25935872,"num_objects":3166,"num_object_clones":0,"num_object_copies":12664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":367,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9661","reported_seq":9783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.841195+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:30.841195+0000","last_peered":"2024-09-11T05:03:30.841195+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:30.841195+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25610560,"num_objects":3127,"num_object_clones":0,"num_object_copies":12508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9658,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9435","reported_seq":9546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.385771+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:31.385771+0000","last_peered":"2024-09-11T05:03:31.385771+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:31.385771+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25334944,"num_objects":3093,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9494","reported_seq":9610,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.922768+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:30.922768+0000","last_peered":"2024-09-11T05:03:30.922768+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:30.922768+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25874160,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9298","reported_seq":13898,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:30.954424+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:30.954424+0000","last_peered":"2024-09-11T05:03:30.954424+0000","last_clean":"2024-09-11T05:03:30.954424+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:30.954424+0000","last_undegraded":"2024-09-11T05:03:30.954424+0000","last_fullsized":"2024-09-11T05:03:30.954424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9298,"log_dups_size":0,"ondisk_log_size":9298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25280512,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3108,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9578","reported_seq":10361,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.593844+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:31.593844+0000","last_peered":"2024-09-11T05:03:31.593844+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:31.593844+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9578,"log_dups_size":0,"ondisk_log_size":9578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762384,"num_objects":3145,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9486","reported_seq":9592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.035035+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:31.035035+0000","last_peered":"2024-09-11T05:03:31.035035+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:31.035035+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9486,"log_dups_size":0,"ondisk_log_size":9486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25441440,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":842}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9407","reported_seq":9525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.556022+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:31.556022+0000","last_peered":"2024-09-11T05:03:31.556022+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:31.556022+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24589472,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9574","reported_seq":9693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.203550+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:29.203550+0000","last_peered":"2024-09-11T05:03:29.203550+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:29.203550+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26236064,"num_objects":3203,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":409506280,"num_objects":49994,"num_object_clones":0,"num_object_copies":199976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135909,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":818921472,"data_stored":818921472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151413,"ondisk_log_size":151413,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705200,"kb_used_data":200092,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93666640,"statfs":{"total":96636764160,"available":95914639360,"internally_reserved":0,"allocated":204894208,"data_stored":204705006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718124,"kb_used_data":199628,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93653716,"statfs":{"total":96636764160,"available":95901405184,"internally_reserved":0,"allocated":204419072,"data_stored":204229870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619232,"kb_used_data":200272,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93752608,"statfs":{"total":96636764160,"available":96002670592,"internally_reserved":0,"allocated":205078528,"data_stored":204889326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":428994258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698704,"kb_used_data":201984,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93673136,"statfs":{"total":96636764160,"available":95921291264,"internally_reserved":0,"allocated":206831616,"data_stored":206642414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,7,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205938688,"data_stored":205938688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204558336,"data_stored":204558336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203988992,"data_stored":203988992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204435456,"data_stored":204435456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:32.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:32.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9412", 'reported_seq': 9505, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:31.008778+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:31.008778+0000', 'last_peered': '2024-09-11T05:03:31.008778+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:31.008778+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9412, 'log_dups_size': 0, 'ondisk_log_size': 9412, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25411584, 'num_objects': 3102, 'num_object_clones': 0, 'num_object_copies': 12408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9412, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9512", 'reported_seq': 9626, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:29.089402+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:29.089402+0000', 'last_peered': '2024-09-11T05:03:29.089402+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:29.089402+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9512, 'log_dups_size': 0, 'ondisk_log_size': 9512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26424480, 'num_objects': 3226, 'num_object_clones': 0, 'num_object_copies': 12904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9510, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 968}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9114", 'reported_seq': 9217, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:31.386920+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:31.386920+0000', 'last_peered': '2024-09-11T05:03:31.386920+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:31.386920+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9114, 'log_dups_size': 0, 'ondisk_log_size': 9114, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24951400, 'num_objects': 3047, 'num_object_clones': 0, 'num_object_copies': 12188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3047, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9113, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 836}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9319", 'reported_seq': 9424, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:29.208926+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:29.208926+0000', 'last_peered': '2024-09-11T05:03:29.208926+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:29.208926+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9319, 'log_dups_size': 0, 'ondisk_log_size': 9319, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25367712, 'num_objects': 3097, 'num_object_clones': 0, 'num_object_copies': 12388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9317, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 836}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9429", 'reported_seq': 9551, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:31.033572+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:31.033572+0000', 'last_peered': '2024-09-11T05:03:31.033572+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:31.033572+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9429, 'log_dups_size': 0, 'ondisk_log_size': 9429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25408672, 'num_objects': 3102, 'num_object_clones': 0, 'num_object_copies': 12408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9427, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 881}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:32.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9493", 'reported_seq': 9598, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:30.754918+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:30.754918+0000', 'last_peered': '2024-09-11T05:03:30.754918+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:30.754918+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9493, 'log_dups_size': 0, 'ondisk_log_size': 9493, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25780224, 'num_objects': 3147, 'num_object_clones': 0, 'num_object_copies': 12588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9493, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9544", 'reported_seq': 9637, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:29.283917+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:29.283917+0000', 'last_peered': '2024-09-11T05:03:29.283917+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:29.283917+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9544, 'log_dups_size': 0, 'ondisk_log_size': 9544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26096800, 'num_objects': 3186, 'num_object_clones': 0, 'num_object_copies': 12744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9542, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 879}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9661", 'reported_seq': 9783, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:30.841195+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:30.841195+0000', 'last_peered': '2024-09-11T05:03:30.841195+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:30.841195+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9661, 'log_dups_size': 0, 'ondisk_log_size': 9661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25610560, 'num_objects': 3127, 'num_object_clones': 0, 'num_object_copies': 12508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9658, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 761}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9435", 'reported_seq': 9546, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:31.385771+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:31.385771+0000', 'last_peered': '2024-09-11T05:03:31.385771+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:31.385771+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9435, 'log_dups_size': 0, 'ondisk_log_size': 9435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25334944, 'num_objects': 3093, 'num_object_clones': 0, 'num_object_copies': 12372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9433, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 787}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9494", 'reported_seq': 9610, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:30.922768+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:30.922768+0000', 'last_peered': '2024-09-11T05:03:30.922768+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:30.922768+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9494, 'log_dups_size': 0, 'ondisk_log_size': 9494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25874160, 'num_objects': 3159, 'num_object_clones': 0, 'num_object_copies': 12636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9491, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 861}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9578", 'reported_seq': 10361, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:31.593844+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:31.593844+0000', 'last_peered': '2024-09-11T05:03:31.593844+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:31.593844+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9578, 'log_dups_size': 0, 'ondisk_log_size': 9578, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25762384, 'num_objects': 3145, 'num_object_clones': 0, 'num_object_copies': 12580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1984, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9577, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 350, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:32.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9486", 'reported_seq': 9592, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:31.035035+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:31.035035+0000', 'last_peered': '2024-09-11T05:03:31.035035+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:31.035035+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9486, 'log_dups_size': 0, 'ondisk_log_size': 9486, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25441440, 'num_objects': 3106, 'num_object_clones': 0, 'num_object_copies': 12424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9484, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 842}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:32.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:32.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9407", 'reported_seq': 9525, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:31.556022+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:31.556022+0000', 'last_peered': '2024-09-11T05:03:31.556022+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:31.556022+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9407, 'log_dups_size': 0, 'ondisk_log_size': 9407, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24589472, 'num_objects': 3002, 'num_object_clones': 0, 'num_object_copies': 12008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:32.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:32.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9574", 'reported_seq': 9693, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:29.203550+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:29.203550+0000', 'last_peered': '2024-09-11T05:03:29.203550+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:29.203550+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9574, 'log_dups_size': 0, 'ondisk_log_size': 9574, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26236064, 'num_objects': 3203, 'num_object_clones': 0, 'num_object_copies': 12812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9572, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:32.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31675/199976 objects degraded (15.839%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":539,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49994,"data_bytes":409506280,"bytes_used":2807050240,"bytes_avail":383740006400,"bytes_total":386547056640,"degraded_objects":31675,"degraded_total":199976,"degraded_ratio":0.15839400728087372,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":238},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:32.913 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:03:32.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:33.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:33.565 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:33.591 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-11T05:03:31.810481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409506280,"num_objects":49994,"num_object_clones":0,"num_object_copies":199976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135909,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151413,"ondisk_log_size":151413,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741260,"kb_used_data":801976,"kb_used_omap":14,"kb_used_meta":1939185,"kb_avail":374746100,"statfs":{"total":386547056640,"available":383740006400,"internally_reserved":0,"allocated":821223424,"data_stored":820466616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":1985726006},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,22,10,17,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23453696,"num_objects":-2863,"num_object_clones":0,"num_object_copies":-11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004882"},"pg_stats":[{"pgid":"1.f","version":"35'9412","reported_seq":9505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.008778+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:31.008778+0000","last_peered":"2024-09-11T05:03:31.008778+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:31.008778+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25411584,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9512","reported_seq":9626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.089402+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:29.089402+0000","last_peered":"2024-09-11T05:03:29.089402+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:29.089402+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424480,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9114","reported_seq":9217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.386920+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:31.386920+0000","last_peered":"2024-09-11T05:03:31.386920+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:31.386920+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9114,"log_dups_size":0,"ondisk_log_size":9114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24951400,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9113,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":836}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9319","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.208926+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:29.208926+0000","last_peered":"2024-09-11T05:03:29.208926+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:29.208926+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9319,"log_dups_size":0,"ondisk_log_size":9319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25367712,"num_objects":3097,"num_object_clones":0,"num_object_copies":12388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9317,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9429","reported_seq":9551,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.033572+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:31.033572+0000","last_peered":"2024-09-11T05:03:31.033572+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:31.033572+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25408672,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9427,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9493","reported_seq":9598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.754918+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:30.754918+0000","last_peered":"2024-09-11T05:03:30.754918+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:30.754918+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25780224,"num_objects":3147,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9544","reported_seq":9637,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.283917+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:29.283917+0000","last_peered":"2024-09-11T05:03:29.283917+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:29.283917+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096800,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":879}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9657","reported_seq":14514,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:30.473813+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:30.473813+0000","last_peered":"2024-09-11T05:03:30.473813+0000","last_clean":"2024-09-11T05:03:30.473813+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:30.473813+0000","last_undegraded":"2024-09-11T05:03:30.473813+0000","last_fullsized":"2024-09-11T05:03:30.473813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9657,"log_dups_size":0,"ondisk_log_size":9657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25935872,"num_objects":3166,"num_object_clones":0,"num_object_copies":12664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":367,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9661","reported_seq":9783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.841195+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:30.841195+0000","last_peered":"2024-09-11T05:03:30.841195+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:30.841195+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25610560,"num_objects":3127,"num_object_clones":0,"num_object_copies":12508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9658,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9435","reported_seq":9546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.385771+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:31.385771+0000","last_peered":"2024-09-11T05:03:31.385771+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:31.385771+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25334944,"num_objects":3093,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9494","reported_seq":9610,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.922768+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:30.922768+0000","last_peered":"2024-09-11T05:03:30.922768+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:30.922768+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25874160,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9298","reported_seq":13898,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:30.954424+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:30.954424+0000","last_peered":"2024-09-11T05:03:30.954424+0000","last_clean":"2024-09-11T05:03:30.954424+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:30.954424+0000","last_undegraded":"2024-09-11T05:03:30.954424+0000","last_fullsized":"2024-09-11T05:03:30.954424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9298,"log_dups_size":0,"ondisk_log_size":9298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25280512,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3108,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9578","reported_seq":10361,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.593844+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:31.593844+0000","last_peered":"2024-09-11T05:03:31.593844+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:31.593844+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9578,"log_dups_size":0,"ondisk_log_size":9578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762384,"num_objects":3145,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9486","reported_seq":9592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.035035+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:31.035035+0000","last_peered":"2024-09-11T05:03:31.035035+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:31.035035+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9486,"log_dups_size":0,"ondisk_log_size":9486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25441440,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":842}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9407","reported_seq":9525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.556022+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:31.556022+0000","last_peered":"2024-09-11T05:03:31.556022+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:31.556022+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24589472,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9574","reported_seq":9693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:29.203550+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:29.203550+0000","last_peered":"2024-09-11T05:03:29.203550+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:29.203550+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26236064,"num_objects":3203,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":915}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":409506280,"num_objects":49994,"num_object_clones":0,"num_object_copies":199976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135909,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":818921472,"data_stored":818921472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151413,"ondisk_log_size":151413,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705200,"kb_used_data":200092,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93666640,"statfs":{"total":96636764160,"available":95914639360,"internally_reserved":0,"allocated":204894208,"data_stored":204705006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718124,"kb_used_data":199628,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93653716,"statfs":{"total":96636764160,"available":95901405184,"internally_reserved":0,"allocated":204419072,"data_stored":204229870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619232,"kb_used_data":200272,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93752608,"statfs":{"total":96636764160,"available":96002670592,"internally_reserved":0,"allocated":205078528,"data_stored":204889326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":428994258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698704,"kb_used_data":201984,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93673136,"statfs":{"total":96636764160,"available":95921291264,"internally_reserved":0,"allocated":206831616,"data_stored":206642414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,7,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205938688,"data_stored":205938688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204558336,"data_stored":204558336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203988992,"data_stored":203988992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204435456,"data_stored":204435456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:36.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:37.266 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:37.290 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-11T05:03:35.811966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":407499240,"num_objects":49749,"num_object_clones":0,"num_object_copies":198996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49749,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136154,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151658,"ondisk_log_size":151658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759540,"kb_used_data":798044,"kb_used_omap":14,"kb_used_meta":1961393,"kb_avail":374727820,"statfs":{"total":386547056640,"available":383721287680,"internally_reserved":0,"allocated":817197056,"data_stored":816440248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2008466998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,20,8,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18808832,"num_objects":-2296,"num_object_clones":0,"num_object_copies":-9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004645"},"pg_stats":[{"pgid":"1.f","version":"35'9412","reported_seq":9505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.008778+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:31.008778+0000","last_peered":"2024-09-11T05:03:31.008778+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:31.008778+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25411584,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9559","reported_seq":9673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.054777+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:34.054777+0000","last_peered":"2024-09-11T05:03:34.054777+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:34.054777+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26039456,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9114","reported_seq":9217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.386920+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:31.386920+0000","last_peered":"2024-09-11T05:03:31.386920+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:31.386920+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9114,"log_dups_size":0,"ondisk_log_size":9114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24951400,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9113,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":836}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9373","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.290856+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:34.290856+0000","last_peered":"2024-09-11T05:03:34.290856+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:34.290856+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9373,"log_dups_size":0,"ondisk_log_size":9373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24925344,"num_objects":3043,"num_object_clones":0,"num_object_copies":12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9371,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9429","reported_seq":9551,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.033572+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:31.033572+0000","last_peered":"2024-09-11T05:03:31.033572+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:31.033572+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25408672,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9427,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9493","reported_seq":9598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.754918+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:30.754918+0000","last_peered":"2024-09-11T05:03:30.754918+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:30.754918+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25780224,"num_objects":3147,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9617","reported_seq":9710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.289816+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:34.289816+0000","last_peered":"2024-09-11T05:03:34.289816+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:34.289816+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498784,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9657","reported_seq":14514,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:30.473813+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:30.473813+0000","last_peered":"2024-09-11T05:03:30.473813+0000","last_clean":"2024-09-11T05:03:30.473813+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:30.473813+0000","last_undegraded":"2024-09-11T05:03:30.473813+0000","last_fullsized":"2024-09-11T05:03:30.473813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9657,"log_dups_size":0,"ondisk_log_size":9657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25935872,"num_objects":3166,"num_object_clones":0,"num_object_copies":12664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":367,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9661","reported_seq":9783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.841195+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:30.841195+0000","last_peered":"2024-09-11T05:03:30.841195+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:30.841195+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25610560,"num_objects":3127,"num_object_clones":0,"num_object_copies":12508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9658,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9435","reported_seq":9546,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.385771+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:31.385771+0000","last_peered":"2024-09-11T05:03:31.385771+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:31.385771+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25334944,"num_objects":3093,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9494","reported_seq":9610,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:30.922768+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:30.922768+0000","last_peered":"2024-09-11T05:03:30.922768+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:30.922768+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25874160,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9298","reported_seq":13898,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:30.954424+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:30.954424+0000","last_peered":"2024-09-11T05:03:30.954424+0000","last_clean":"2024-09-11T05:03:30.954424+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:30.954424+0000","last_undegraded":"2024-09-11T05:03:30.954424+0000","last_fullsized":"2024-09-11T05:03:30.954424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9298,"log_dups_size":0,"ondisk_log_size":9298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25280512,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3108,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9578","reported_seq":10361,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.593844+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:31.593844+0000","last_peered":"2024-09-11T05:03:31.593844+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:31.593844+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9578,"log_dups_size":0,"ondisk_log_size":9578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762384,"num_objects":3145,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1161}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9486","reported_seq":9592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.035035+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:31.035035+0000","last_peered":"2024-09-11T05:03:31.035035+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:31.035035+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9486,"log_dups_size":0,"ondisk_log_size":9486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25441440,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":842}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9407","reported_seq":9525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:31.556022+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:31.556022+0000","last_peered":"2024-09-11T05:03:31.556022+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:31.556022+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24589472,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9645","reported_seq":9764,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.294109+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:34.294109+0000","last_peered":"2024-09-11T05:03:34.294109+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:34.294109+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25654432,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":407499240,"num_objects":49749,"num_object_clones":0,"num_object_copies":198996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49749,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136154,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":814813184,"data_stored":814813184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151658,"ondisk_log_size":151658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705200,"kb_used_data":200092,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93666640,"statfs":{"total":96636764160,"available":95914639360,"internally_reserved":0,"allocated":204894208,"data_stored":204705006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718124,"kb_used_data":199628,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93653716,"statfs":{"total":96636764160,"available":95901405184,"internally_reserved":0,"allocated":204419072,"data_stored":204229870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619232,"kb_used_data":200272,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93752608,"statfs":{"total":96636764160,"available":96002670592,"internally_reserved":0,"allocated":205078528,"data_stored":204889326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":428994258},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716984,"kb_used_data":198052,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93654856,"statfs":{"total":96636764160,"available":95902572544,"internally_reserved":0,"allocated":202805248,"data_stored":202616046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531361487},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":201830400,"data_stored":201830400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204558336,"data_stored":204558336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203988992,"data_stored":203988992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204435456,"data_stored":204435456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:37.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:37.919 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:37.920 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:37.938 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-11T05:03:37.812680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":401224168,"num_objects":48983,"num_object_clones":0,"num_object_copies":195932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136920,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152424,"ondisk_log_size":152424,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778052,"kb_used_data":786044,"kb_used_omap":14,"kb_used_meta":1991921,"kb_avail":374709308,"statfs":{"total":386547056640,"available":383702331392,"internally_reserved":0,"allocated":804909056,"data_stored":804152248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2039727670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,10,10,11,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23142400,"num_objects":-2825,"num_object_clones":0,"num_object_copies":-11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004732"},"pg_stats":[{"pgid":"1.f","version":"35'9475","reported_seq":9569,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.014459+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:36.014459+0000","last_peered":"2024-09-11T05:03:36.014459+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:36.014459+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9475,"log_dups_size":0,"ondisk_log_size":9475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24895488,"num_objects":3039,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9475,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9559","reported_seq":9673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.054777+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:34.054777+0000","last_peered":"2024-09-11T05:03:34.054777+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:34.054777+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26039456,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9174","reported_seq":9278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.589847+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:36.589847+0000","last_peered":"2024-09-11T05:03:36.589847+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:36.589847+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24459880,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9173,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9373","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.290856+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:34.290856+0000","last_peered":"2024-09-11T05:03:34.290856+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:34.290856+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9373,"log_dups_size":0,"ondisk_log_size":9373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24925344,"num_objects":3043,"num_object_clones":0,"num_object_copies":12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9371,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9493","reported_seq":9612,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.050918+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:36.050918+0000","last_peered":"2024-09-11T05:03:36.050918+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:36.050918+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884384,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9563","reported_seq":9668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:35.992670+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:35.992670+0000","last_peered":"2024-09-11T05:03:35.992670+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:35.992670+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25206784,"num_objects":3077,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":783}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9617","reported_seq":9710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.289816+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:34.289816+0000","last_peered":"2024-09-11T05:03:34.289816+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:34.289816+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498784,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9727","reported_seq":14584,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:35.924663+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:35.924663+0000","last_peered":"2024-09-11T05:03:35.924663+0000","last_clean":"2024-09-11T05:03:35.924663+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:35.924663+0000","last_undegraded":"2024-09-11T05:03:35.924663+0000","last_fullsized":"2024-09-11T05:03:35.924663+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9727,"log_dups_size":0,"ondisk_log_size":9727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25362432,"num_objects":3096,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9722","reported_seq":9844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:35.951963+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:35.951963+0000","last_peered":"2024-09-11T05:03:35.951963+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:35.951963+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25110848,"num_objects":3066,"num_object_clones":0,"num_object_copies":12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9490","reported_seq":9602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.347739+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:36.347739+0000","last_peered":"2024-09-11T05:03:36.347739+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:36.347739+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9490,"log_dups_size":0,"ondisk_log_size":9490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884384,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":732}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9554","reported_seq":9670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:35.961928+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:35.961928+0000","last_peered":"2024-09-11T05:03:35.961928+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:35.961928+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9554,"log_dups_size":0,"ondisk_log_size":9554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25382640,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9369","reported_seq":13969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:36.073203+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:36.073203+0000","last_peered":"2024-09-11T05:03:36.073203+0000","last_clean":"2024-09-11T05:03:36.073203+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:36.073203+0000","last_undegraded":"2024-09-11T05:03:36.073203+0000","last_fullsized":"2024-09-11T05:03:36.073203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24698880,"num_objects":3015,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3179,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9630","reported_seq":10557,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.654548+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:36.654548+0000","last_peered":"2024-09-11T05:03:36.654548+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:36.654548+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9630,"log_dups_size":0,"ondisk_log_size":9630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25336400,"num_objects":3093,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9548","reported_seq":9654,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:35.830882+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:35.830882+0000","last_peered":"2024-09-11T05:03:35.830882+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:35.830882+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24933536,"num_objects":3044,"num_object_clones":0,"num_object_copies":12176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":780}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9485","reported_seq":9603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.608671+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:36.608671+0000","last_peered":"2024-09-11T05:03:36.608671+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:36.608671+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23950496,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9645","reported_seq":9764,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.294109+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:34.294109+0000","last_peered":"2024-09-11T05:03:34.294109+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:34.294109+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25654432,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":401224168,"num_objects":48983,"num_object_clones":0,"num_object_copies":195932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136920,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802492416,"data_stored":802492416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152424,"ondisk_log_size":152424,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723460,"kb_used_data":196148,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93648380,"statfs":{"total":96636764160,"available":95895941120,"internally_reserved":0,"allocated":200855552,"data_stored":200666350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":539946704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,8,0,5,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718252,"kb_used_data":195608,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93653588,"statfs":{"total":96636764160,"available":95901274112,"internally_reserved":0,"allocated":200302592,"data_stored":200113390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619356,"kb_used_data":196236,"kb_used_omap":4,"kb_used_meta":423099,"kb_avail":93752484,"statfs":{"total":96636764160,"available":96002543616,"internally_reserved":0,"allocated":200945664,"data_stored":200756462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":433254098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716984,"kb_used_data":198052,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93654856,"statfs":{"total":96636764160,"available":95902572544,"internally_reserved":0,"allocated":202805248,"data_stored":202616046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531361487},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":201830400,"data_stored":201830400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200417280,"data_stored":200417280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199917568,"data_stored":199917568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200327168,"data_stored":200327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:37.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:37.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9475", 'reported_seq': 9569, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:36.014459+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:36.014459+0000', 'last_peered': '2024-09-11T05:03:36.014459+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:36.014459+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9475, 'log_dups_size': 0, 'ondisk_log_size': 9475, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24895488, 'num_objects': 3039, 'num_object_clones': 0, 'num_object_copies': 12156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9475, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 748}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:37.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:37.939 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9559", 'reported_seq': 9673, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:34.054777+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:34.054777+0000', 'last_peered': '2024-09-11T05:03:34.054777+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:34.054777+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9559, 'log_dups_size': 0, 'ondisk_log_size': 9559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26039456, 'num_objects': 3179, 'num_object_clones': 0, 'num_object_copies': 12716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9557, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 921}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:37.939 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:37.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9174", 'reported_seq': 9278, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:36.589847+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:36.589847+0000', 'last_peered': '2024-09-11T05:03:36.589847+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:36.589847+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9174, 'log_dups_size': 0, 'ondisk_log_size': 9174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24459880, 'num_objects': 2987, 'num_object_clones': 0, 'num_object_copies': 11948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2987, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9173, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 776}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:37.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:37.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9373", 'reported_seq': 9481, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:34.290856+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:34.290856+0000', 'last_peered': '2024-09-11T05:03:34.290856+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:34.290856+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9373, 'log_dups_size': 0, 'ondisk_log_size': 9373, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24925344, 'num_objects': 3043, 'num_object_clones': 0, 'num_object_copies': 12172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9371, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 782}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:37.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:37.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9493", 'reported_seq': 9612, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:36.050918+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:36.050918+0000', 'last_peered': '2024-09-11T05:03:36.050918+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:36.050918+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9493, 'log_dups_size': 0, 'ondisk_log_size': 9493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24884384, 'num_objects': 3038, 'num_object_clones': 0, 'num_object_copies': 12152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9491, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 817}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:37.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:37.940 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9563", 'reported_seq': 9668, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:35.992670+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:35.992670+0000', 'last_peered': '2024-09-11T05:03:35.992670+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:35.992670+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9563, 'log_dups_size': 0, 'ondisk_log_size': 9563, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25206784, 'num_objects': 3077, 'num_object_clones': 0, 'num_object_copies': 12308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9563, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 783}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9617", 'reported_seq': 9710, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:34.289816+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:34.289816+0000', 'last_peered': '2024-09-11T05:03:34.289816+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:34.289816+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9617, 'log_dups_size': 0, 'ondisk_log_size': 9617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25498784, 'num_objects': 3113, 'num_object_clones': 0, 'num_object_copies': 12452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9615, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 806}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9722", 'reported_seq': 9844, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:35.951963+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:35.951963+0000', 'last_peered': '2024-09-11T05:03:35.951963+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:35.951963+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9722, 'log_dups_size': 0, 'ondisk_log_size': 9722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25110848, 'num_objects': 3066, 'num_object_clones': 0, 'num_object_copies': 12264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9719, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 700}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9490", 'reported_seq': 9602, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:36.347739+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:36.347739+0000', 'last_peered': '2024-09-11T05:03:36.347739+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:36.347739+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9490, 'log_dups_size': 0, 'ondisk_log_size': 9490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24884384, 'num_objects': 3038, 'num_object_clones': 0, 'num_object_copies': 12152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9488, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 732}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9554", 'reported_seq': 9670, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:35.961928+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:35.961928+0000', 'last_peered': '2024-09-11T05:03:35.961928+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:35.961928+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9554, 'log_dups_size': 0, 'ondisk_log_size': 9554, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25382640, 'num_objects': 3099, 'num_object_clones': 0, 'num_object_copies': 12396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9551, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 801}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:37.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:37.942 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9630", 'reported_seq': 10557, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:36.654548+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:36.654548+0000', 'last_peered': '2024-09-11T05:03:36.654548+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:36.654548+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9630, 'log_dups_size': 0, 'ondisk_log_size': 9630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25336400, 'num_objects': 3093, 'num_object_clones': 0, 'num_object_copies': 12372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1912, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9629, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 422, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1181}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:37.942 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:37.942 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9548", 'reported_seq': 9654, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:35.830882+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:35.830882+0000', 'last_peered': '2024-09-11T05:03:35.830882+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:35.830882+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9548, 'log_dups_size': 0, 'ondisk_log_size': 9548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24933536, 'num_objects': 3044, 'num_object_clones': 0, 'num_object_copies': 12176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9546, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 780}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:37.942 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:37.942 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9485", 'reported_seq': 9603, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:36.608671+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:36.608671+0000', 'last_peered': '2024-09-11T05:03:36.608671+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:36.608671+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9485, 'log_dups_size': 0, 'ondisk_log_size': 9485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23950496, 'num_objects': 2924, 'num_object_clones': 0, 'num_object_copies': 11696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9483, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 621}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:37.942 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:37.942 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9645", 'reported_seq': 9764, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:34.294109+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:34.294109+0000', 'last_peered': '2024-09-11T05:03:34.294109+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:34.294109+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9645, 'log_dups_size': 0, 'ondisk_log_size': 9645, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25654432, 'num_objects': 3132, 'num_object_clones': 0, 'num_object_copies': 12528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9643, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:37.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:38.734 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31603/195932 objects degraded (16.130%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":545,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":48983,"data_bytes":401224168,"bytes_used":2844725248,"bytes_avail":383702331392,"bytes_total":386547056640,"degraded_objects":31603,"degraded_total":195932,"degraded_ratio":0.16129575567033461,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":235},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:38.752 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:03:38.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:39.408 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:39.430 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-11T05:03:37.812680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":401224168,"num_objects":48983,"num_object_clones":0,"num_object_copies":195932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136920,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152424,"ondisk_log_size":152424,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778052,"kb_used_data":786044,"kb_used_omap":14,"kb_used_meta":1991921,"kb_avail":374709308,"statfs":{"total":386547056640,"available":383702331392,"internally_reserved":0,"allocated":804909056,"data_stored":804152248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2039727670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,10,10,11,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23142400,"num_objects":-2825,"num_object_clones":0,"num_object_copies":-11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004732"},"pg_stats":[{"pgid":"1.f","version":"35'9475","reported_seq":9569,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.014459+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:36.014459+0000","last_peered":"2024-09-11T05:03:36.014459+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:36.014459+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9475,"log_dups_size":0,"ondisk_log_size":9475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24895488,"num_objects":3039,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9475,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9559","reported_seq":9673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.054777+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:34.054777+0000","last_peered":"2024-09-11T05:03:34.054777+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:34.054777+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26039456,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9174","reported_seq":9278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.589847+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:36.589847+0000","last_peered":"2024-09-11T05:03:36.589847+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:36.589847+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24459880,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9173,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9373","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.290856+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:34.290856+0000","last_peered":"2024-09-11T05:03:34.290856+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:34.290856+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9373,"log_dups_size":0,"ondisk_log_size":9373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24925344,"num_objects":3043,"num_object_clones":0,"num_object_copies":12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9371,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9493","reported_seq":9612,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.050918+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:36.050918+0000","last_peered":"2024-09-11T05:03:36.050918+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:36.050918+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884384,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9563","reported_seq":9668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:35.992670+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:35.992670+0000","last_peered":"2024-09-11T05:03:35.992670+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:35.992670+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9563,"log_dups_size":0,"ondisk_log_size":9563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25206784,"num_objects":3077,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":783}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9617","reported_seq":9710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.289816+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:34.289816+0000","last_peered":"2024-09-11T05:03:34.289816+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:34.289816+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498784,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9727","reported_seq":14584,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:35.924663+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:35.924663+0000","last_peered":"2024-09-11T05:03:35.924663+0000","last_clean":"2024-09-11T05:03:35.924663+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:35.924663+0000","last_undegraded":"2024-09-11T05:03:35.924663+0000","last_fullsized":"2024-09-11T05:03:35.924663+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9727,"log_dups_size":0,"ondisk_log_size":9727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25362432,"num_objects":3096,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9722","reported_seq":9844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:35.951963+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:35.951963+0000","last_peered":"2024-09-11T05:03:35.951963+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:35.951963+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25110848,"num_objects":3066,"num_object_clones":0,"num_object_copies":12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9490","reported_seq":9602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.347739+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:36.347739+0000","last_peered":"2024-09-11T05:03:36.347739+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:36.347739+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9490,"log_dups_size":0,"ondisk_log_size":9490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884384,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":732}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9554","reported_seq":9670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:35.961928+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:35.961928+0000","last_peered":"2024-09-11T05:03:35.961928+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:35.961928+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9554,"log_dups_size":0,"ondisk_log_size":9554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25382640,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9369","reported_seq":13969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:36.073203+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:36.073203+0000","last_peered":"2024-09-11T05:03:36.073203+0000","last_clean":"2024-09-11T05:03:36.073203+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:36.073203+0000","last_undegraded":"2024-09-11T05:03:36.073203+0000","last_fullsized":"2024-09-11T05:03:36.073203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24698880,"num_objects":3015,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3179,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9630","reported_seq":10557,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.654548+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:36.654548+0000","last_peered":"2024-09-11T05:03:36.654548+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:36.654548+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9630,"log_dups_size":0,"ondisk_log_size":9630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25336400,"num_objects":3093,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9548","reported_seq":9654,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:35.830882+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:35.830882+0000","last_peered":"2024-09-11T05:03:35.830882+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:35.830882+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24933536,"num_objects":3044,"num_object_clones":0,"num_object_copies":12176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":780}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9485","reported_seq":9603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:36.608671+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:36.608671+0000","last_peered":"2024-09-11T05:03:36.608671+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:36.608671+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23950496,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9645","reported_seq":9764,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:34.294109+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:34.294109+0000","last_peered":"2024-09-11T05:03:34.294109+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:34.294109+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25654432,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":401224168,"num_objects":48983,"num_object_clones":0,"num_object_copies":195932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136920,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802492416,"data_stored":802492416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152424,"ondisk_log_size":152424,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723460,"kb_used_data":196148,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93648380,"statfs":{"total":96636764160,"available":95895941120,"internally_reserved":0,"allocated":200855552,"data_stored":200666350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":539946704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,8,0,5,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718252,"kb_used_data":195608,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93653588,"statfs":{"total":96636764160,"available":95901274112,"internally_reserved":0,"allocated":200302592,"data_stored":200113390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619356,"kb_used_data":196236,"kb_used_omap":4,"kb_used_meta":423099,"kb_avail":93752484,"statfs":{"total":96636764160,"available":96002543616,"internally_reserved":0,"allocated":200945664,"data_stored":200756462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":433254098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716984,"kb_used_data":198052,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93654856,"statfs":{"total":96636764160,"available":95902572544,"internally_reserved":0,"allocated":202805248,"data_stored":202616046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531361487},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":201830400,"data_stored":201830400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200417280,"data_stored":200417280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199917568,"data_stored":199917568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200327168,"data_stored":200327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:42.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:43.120 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:43.120 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:43.137 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-09-11T05:03:41.814405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":393253352,"num_objects":48010,"num_object_clones":0,"num_object_copies":192040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153397,"ondisk_log_size":153397,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763704,"kb_used_data":770276,"kb_used_omap":14,"kb_used_meta":1993329,"kb_avail":374723656,"statfs":{"total":386547056640,"available":383717023744,"internally_reserved":0,"allocated":788762624,"data_stored":788005816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2041169462},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,8,4,8,14,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22781952,"num_objects":-2781,"num_object_clones":0,"num_object_copies":-11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2781,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004785"},"pg_stats":[{"pgid":"1.f","version":"35'9523","reported_seq":9617,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:40.773942+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:40.773942+0000","last_peered":"2024-09-11T05:03:40.773942+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:40.773942+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9523,"log_dups_size":0,"ondisk_log_size":9523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24502272,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9634","reported_seq":9748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.336741+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:39.336741+0000","last_peered":"2024-09-11T05:03:39.336741+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:39.336741+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9634,"log_dups_size":0,"ondisk_log_size":9634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25425056,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9632,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9235","reported_seq":9339,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.334108+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:41.334108+0000","last_peered":"2024-09-11T05:03:41.334108+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:41.334108+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9235,"log_dups_size":0,"ondisk_log_size":9235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23960168,"num_objects":2926,"num_object_clones":0,"num_object_copies":11704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9234,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":715}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9433","reported_seq":9541,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.242124+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:39.242124+0000","last_peered":"2024-09-11T05:03:39.242124+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:39.242124+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9433,"log_dups_size":0,"ondisk_log_size":9433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24433824,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":722}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9549","reported_seq":9672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.080441+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:41.080441+0000","last_peered":"2024-09-11T05:03:41.080441+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:41.080441+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9549,"log_dups_size":0,"ondisk_log_size":9549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24425632,"num_objects":2982,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9623","reported_seq":9728,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:40.994148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:40.994148+0000","last_peered":"2024-09-11T05:03:40.994148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:40.994148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9623,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9661","reported_seq":9754,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.287003+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:39.287003+0000","last_peered":"2024-09-11T05:03:39.287003+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:39.287003+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138336,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":762}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9799","reported_seq":14656,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:40.988458+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:40.988458+0000","last_peered":"2024-09-11T05:03:40.988458+0000","last_clean":"2024-09-11T05:03:40.988458+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:40.988458+0000","last_undegraded":"2024-09-11T05:03:40.988458+0000","last_fullsized":"2024-09-11T05:03:40.988458+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9799,"log_dups_size":0,"ondisk_log_size":9799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":3024,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9789","reported_seq":9911,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.037907+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:41.037907+0000","last_peered":"2024-09-11T05:03:41.037907+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:41.037907+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9789,"log_dups_size":0,"ondisk_log_size":9789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24561984,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9551","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.688407+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:41.688407+0000","last_peered":"2024-09-11T05:03:41.688407+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:41.688407+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24384672,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":671}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9622","reported_seq":9738,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.043188+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:41.043188+0000","last_peered":"2024-09-11T05:03:41.043188+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:41.043188+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24825584,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9619,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9423","reported_seq":14022,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:41.078514+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:41.078514+0000","last_peered":"2024-09-11T05:03:41.078514+0000","last_clean":"2024-09-11T05:03:41.078514+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:41.078514+0000","last_undegraded":"2024-09-11T05:03:41.078514+0000","last_fullsized":"2024-09-11T05:03:41.078514+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24256512,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9703","reported_seq":10708,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.629979+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:41.629979+0000","last_peered":"2024-09-11T05:03:41.629979+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:41.629979+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24738384,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9605","reported_seq":9711,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.079074+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:41.079074+0000","last_peered":"2024-09-11T05:03:41.079074+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:41.079074+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466592,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9603,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":723}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9544","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.418553+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:41.418553+0000","last_peered":"2024-09-11T05:03:41.418553+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:41.418553+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23467168,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9703","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:38.733393+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:38.733393+0000","last_peered":"2024-09-11T05:03:38.733393+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:38.733393+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25179296,"num_objects":3074,"num_object_clones":0,"num_object_copies":12296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9701,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":393253352,"num_objects":48010,"num_object_clones":0,"num_object_copies":192040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":786378752,"data_stored":786378752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153397,"ondisk_log_size":153397,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707012,"kb_used_data":192168,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93664828,"statfs":{"total":96636764160,"available":95912783872,"internally_reserved":0,"allocated":196780032,"data_stored":196590830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":527167184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,2,2,4,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718464,"kb_used_data":191664,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93653376,"statfs":{"total":96636764160,"available":95901057024,"internally_reserved":0,"allocated":196263936,"data_stored":196074734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621008,"kb_used_data":192320,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93750832,"statfs":{"total":96636764160,"available":96000851968,"internally_reserved":0,"allocated":196935680,"data_stored":196746478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":438955730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717220,"kb_used_data":194124,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93654620,"statfs":{"total":96636764160,"available":95902330880,"internally_reserved":0,"allocated":198782976,"data_stored":198593774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535621327},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,4,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197808128,"data_stored":197808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196386816,"data_stored":196386816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195854336,"data_stored":195854336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":196329472,"data_stored":196329472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:43.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:43.751 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:43.781 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-09-11T05:03:41.814405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":393253352,"num_objects":48010,"num_object_clones":0,"num_object_copies":192040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153397,"ondisk_log_size":153397,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763704,"kb_used_data":770276,"kb_used_omap":14,"kb_used_meta":1993329,"kb_avail":374723656,"statfs":{"total":386547056640,"available":383717023744,"internally_reserved":0,"allocated":788762624,"data_stored":788005816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2041169462},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,8,4,8,14,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22781952,"num_objects":-2781,"num_object_clones":0,"num_object_copies":-11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2781,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004785"},"pg_stats":[{"pgid":"1.f","version":"35'9523","reported_seq":9617,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:40.773942+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:40.773942+0000","last_peered":"2024-09-11T05:03:40.773942+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:40.773942+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9523,"log_dups_size":0,"ondisk_log_size":9523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24502272,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9634","reported_seq":9748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.336741+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:39.336741+0000","last_peered":"2024-09-11T05:03:39.336741+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:39.336741+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9634,"log_dups_size":0,"ondisk_log_size":9634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25425056,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9632,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9235","reported_seq":9339,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.334108+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:41.334108+0000","last_peered":"2024-09-11T05:03:41.334108+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:41.334108+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9235,"log_dups_size":0,"ondisk_log_size":9235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23960168,"num_objects":2926,"num_object_clones":0,"num_object_copies":11704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9234,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":715}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9433","reported_seq":9541,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.242124+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:39.242124+0000","last_peered":"2024-09-11T05:03:39.242124+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:39.242124+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9433,"log_dups_size":0,"ondisk_log_size":9433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24433824,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":722}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9549","reported_seq":9672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.080441+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:41.080441+0000","last_peered":"2024-09-11T05:03:41.080441+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:41.080441+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9549,"log_dups_size":0,"ondisk_log_size":9549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24425632,"num_objects":2982,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9623","reported_seq":9728,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:40.994148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:40.994148+0000","last_peered":"2024-09-11T05:03:40.994148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:40.994148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9623,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9661","reported_seq":9754,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.287003+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:39.287003+0000","last_peered":"2024-09-11T05:03:39.287003+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:39.287003+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138336,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":762}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9799","reported_seq":14656,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:40.988458+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:40.988458+0000","last_peered":"2024-09-11T05:03:40.988458+0000","last_clean":"2024-09-11T05:03:40.988458+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:40.988458+0000","last_undegraded":"2024-09-11T05:03:40.988458+0000","last_fullsized":"2024-09-11T05:03:40.988458+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9799,"log_dups_size":0,"ondisk_log_size":9799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":3024,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9789","reported_seq":9911,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.037907+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:41.037907+0000","last_peered":"2024-09-11T05:03:41.037907+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:41.037907+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9789,"log_dups_size":0,"ondisk_log_size":9789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24561984,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9551","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.688407+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:41.688407+0000","last_peered":"2024-09-11T05:03:41.688407+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:41.688407+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24384672,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":671}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9622","reported_seq":9738,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.043188+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:41.043188+0000","last_peered":"2024-09-11T05:03:41.043188+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:41.043188+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24825584,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9619,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9423","reported_seq":14022,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:41.078514+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:41.078514+0000","last_peered":"2024-09-11T05:03:41.078514+0000","last_clean":"2024-09-11T05:03:41.078514+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:41.078514+0000","last_undegraded":"2024-09-11T05:03:41.078514+0000","last_fullsized":"2024-09-11T05:03:41.078514+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24256512,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9703","reported_seq":10708,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.629979+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:41.629979+0000","last_peered":"2024-09-11T05:03:41.629979+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:41.629979+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24738384,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9605","reported_seq":9711,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.079074+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:41.079074+0000","last_peered":"2024-09-11T05:03:41.079074+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:41.079074+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466592,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9603,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":723}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9544","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.418553+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:41.418553+0000","last_peered":"2024-09-11T05:03:41.418553+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:41.418553+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23467168,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9703","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:38.733393+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:38.733393+0000","last_peered":"2024-09-11T05:03:38.733393+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:38.733393+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25179296,"num_objects":3074,"num_object_clones":0,"num_object_copies":12296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9701,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":393253352,"num_objects":48010,"num_object_clones":0,"num_object_copies":192040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":786378752,"data_stored":786378752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153397,"ondisk_log_size":153397,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707012,"kb_used_data":192168,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93664828,"statfs":{"total":96636764160,"available":95912783872,"internally_reserved":0,"allocated":196780032,"data_stored":196590830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":527167184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,2,2,4,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718464,"kb_used_data":191664,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93653376,"statfs":{"total":96636764160,"available":95901057024,"internally_reserved":0,"allocated":196263936,"data_stored":196074734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621008,"kb_used_data":192320,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93750832,"statfs":{"total":96636764160,"available":96000851968,"internally_reserved":0,"allocated":196935680,"data_stored":196746478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":438955730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717220,"kb_used_data":194124,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93654620,"statfs":{"total":96636764160,"available":95902330880,"internally_reserved":0,"allocated":198782976,"data_stored":198593774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535621327},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,4,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197808128,"data_stored":197808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196386816,"data_stored":196386816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195854336,"data_stored":195854336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":196329472,"data_stored":196329472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:43.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:43.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9523", 'reported_seq': 9617, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:40.773942+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:40.773942+0000', 'last_peered': '2024-09-11T05:03:40.773942+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:40.773942+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9523, 'log_dups_size': 0, 'ondisk_log_size': 9523, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24502272, 'num_objects': 2991, 'num_object_clones': 0, 'num_object_copies': 11964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9523, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 700}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:43.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:43.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9634", 'reported_seq': 9748, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:39.336741+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:39.336741+0000', 'last_peered': '2024-09-11T05:03:39.336741+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:39.336741+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9634, 'log_dups_size': 0, 'ondisk_log_size': 9634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25425056, 'num_objects': 3104, 'num_object_clones': 0, 'num_object_copies': 12416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9632, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 846}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:43.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:43.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9235", 'reported_seq': 9339, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:41.334108+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:41.334108+0000', 'last_peered': '2024-09-11T05:03:41.334108+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:41.334108+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9235, 'log_dups_size': 0, 'ondisk_log_size': 9235, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23960168, 'num_objects': 2926, 'num_object_clones': 0, 'num_object_copies': 11704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2926, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9234, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 715}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9433", 'reported_seq': 9541, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:39.242124+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:39.242124+0000', 'last_peered': '2024-09-11T05:03:39.242124+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:39.242124+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9433, 'log_dups_size': 0, 'ondisk_log_size': 9433, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24433824, 'num_objects': 2983, 'num_object_clones': 0, 'num_object_copies': 11932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9431, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 722}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9549", 'reported_seq': 9672, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:41.080441+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:41.080441+0000', 'last_peered': '2024-09-11T05:03:41.080441+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:41.080441+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9549, 'log_dups_size': 0, 'ondisk_log_size': 9549, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24425632, 'num_objects': 2982, 'num_object_clones': 0, 'num_object_copies': 11928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9547, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 761}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9623", 'reported_seq': 9728, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:40.994148+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:40.994148+0000', 'last_peered': '2024-09-11T05:03:40.994148+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:40.994148+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9623, 'log_dups_size': 0, 'ondisk_log_size': 9623, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24715264, 'num_objects': 3017, 'num_object_clones': 0, 'num_object_copies': 12068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9623, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 723}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9661", 'reported_seq': 9754, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:39.287003+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:39.287003+0000', 'last_peered': '2024-09-11T05:03:39.287003+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:39.287003+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9661, 'log_dups_size': 0, 'ondisk_log_size': 9661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25138336, 'num_objects': 3069, 'num_object_clones': 0, 'num_object_copies': 12276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9659, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 762}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:43.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9789", 'reported_seq': 9911, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:41.037907+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:41.037907+0000', 'last_peered': '2024-09-11T05:03:41.037907+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:41.037907+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9789, 'log_dups_size': 0, 'ondisk_log_size': 9789, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24561984, 'num_objects': 2999, 'num_object_clones': 0, 'num_object_copies': 11996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9786, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 633}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9551", 'reported_seq': 9662, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:41.688407+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:41.688407+0000', 'last_peered': '2024-09-11T05:03:41.688407+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:41.688407+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9551, 'log_dups_size': 0, 'ondisk_log_size': 9551, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24384672, 'num_objects': 2977, 'num_object_clones': 0, 'num_object_copies': 11908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9549, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 671}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9622", 'reported_seq': 9738, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:41.043188+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:41.043188+0000', 'last_peered': '2024-09-11T05:03:41.043188+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:41.043188+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9622, 'log_dups_size': 0, 'ondisk_log_size': 9622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24825584, 'num_objects': 3031, 'num_object_clones': 0, 'num_object_copies': 12124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9619, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 733}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9703", 'reported_seq': 10708, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:41.629979+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:41.629979+0000', 'last_peered': '2024-09-11T05:03:41.629979+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:41.629979+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9703, 'log_dups_size': 0, 'ondisk_log_size': 9703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24738384, 'num_objects': 3020, 'num_object_clones': 0, 'num_object_copies': 12080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1873, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9702, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 461, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9605", 'reported_seq': 9711, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:41.079074+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:41.079074+0000', 'last_peered': '2024-09-11T05:03:41.079074+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:41.079074+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9605, 'log_dups_size': 0, 'ondisk_log_size': 9605, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24466592, 'num_objects': 2987, 'num_object_clones': 0, 'num_object_copies': 11948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9603, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 723}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9544", 'reported_seq': 9662, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:41.418553+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:41.418553+0000', 'last_peered': '2024-09-11T05:03:41.418553+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:41.418553+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9544, 'log_dups_size': 0, 'ondisk_log_size': 9544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23467168, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9542, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 562}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:43.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:43.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9703", 'reported_seq': 9822, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:38.733393+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:38.733393+0000', 'last_peered': '2024-09-11T05:03:38.733393+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:38.733393+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9703, 'log_dups_size': 0, 'ondisk_log_size': 9703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25179296, 'num_objects': 3074, 'num_object_clones': 0, 'num_object_copies': 12296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9701, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:43.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:44.611 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31564/192040 objects degraded (16.436%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":551,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":48010,"data_bytes":393253352,"bytes_used":2830032896,"bytes_avail":383717023744,"bytes_total":386547056640,"degraded_objects":31564,"degraded_total":192040,"degraded_ratio":0.1643615913351385,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":165},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:44.634 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:03:44.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:45.405 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-11T05:03:43.815301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":393253352,"num_objects":48010,"num_object_clones":0,"num_object_copies":192040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153397,"ondisk_log_size":153397,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763704,"kb_used_data":770276,"kb_used_omap":14,"kb_used_meta":1993329,"kb_avail":374723656,"statfs":{"total":386547056640,"available":383717023744,"internally_reserved":0,"allocated":788762624,"data_stored":788005816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2041169462},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,8,4,8,14,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16252928,"num_objects":-1984,"num_object_clones":0,"num_object_copies":-7936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1984,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004820"},"pg_stats":[{"pgid":"1.f","version":"35'9523","reported_seq":9617,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:40.773942+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:40.773942+0000","last_peered":"2024-09-11T05:03:40.773942+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:40.773942+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9523,"log_dups_size":0,"ondisk_log_size":9523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24502272,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9634","reported_seq":9748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.336741+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:39.336741+0000","last_peered":"2024-09-11T05:03:39.336741+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:39.336741+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9634,"log_dups_size":0,"ondisk_log_size":9634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25425056,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9632,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9235","reported_seq":9339,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.334108+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:41.334108+0000","last_peered":"2024-09-11T05:03:41.334108+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:41.334108+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9235,"log_dups_size":0,"ondisk_log_size":9235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23960168,"num_objects":2926,"num_object_clones":0,"num_object_copies":11704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9234,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":715}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9433","reported_seq":9541,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.242124+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:39.242124+0000","last_peered":"2024-09-11T05:03:39.242124+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:39.242124+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9433,"log_dups_size":0,"ondisk_log_size":9433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24433824,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":722}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9549","reported_seq":9672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.080441+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:41.080441+0000","last_peered":"2024-09-11T05:03:41.080441+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:41.080441+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9549,"log_dups_size":0,"ondisk_log_size":9549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24425632,"num_objects":2982,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9623","reported_seq":9728,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:40.994148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:40.994148+0000","last_peered":"2024-09-11T05:03:40.994148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:40.994148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9623,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9661","reported_seq":9754,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:39.287003+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:39.287003+0000","last_peered":"2024-09-11T05:03:39.287003+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:39.287003+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138336,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":762}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9799","reported_seq":14656,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:40.988458+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:40.988458+0000","last_peered":"2024-09-11T05:03:40.988458+0000","last_clean":"2024-09-11T05:03:40.988458+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:40.988458+0000","last_undegraded":"2024-09-11T05:03:40.988458+0000","last_fullsized":"2024-09-11T05:03:40.988458+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9799,"log_dups_size":0,"ondisk_log_size":9799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":3024,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9789","reported_seq":9911,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.037907+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:41.037907+0000","last_peered":"2024-09-11T05:03:41.037907+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:41.037907+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9789,"log_dups_size":0,"ondisk_log_size":9789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24561984,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9551","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.688407+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:41.688407+0000","last_peered":"2024-09-11T05:03:41.688407+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:41.688407+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24384672,"num_objects":2977,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":671}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9622","reported_seq":9738,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.043188+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:41.043188+0000","last_peered":"2024-09-11T05:03:41.043188+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:41.043188+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24825584,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9619,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9423","reported_seq":14022,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:41.078514+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:41.078514+0000","last_peered":"2024-09-11T05:03:41.078514+0000","last_clean":"2024-09-11T05:03:41.078514+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:41.078514+0000","last_undegraded":"2024-09-11T05:03:41.078514+0000","last_fullsized":"2024-09-11T05:03:41.078514+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24256512,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9703","reported_seq":10708,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.629979+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:41.629979+0000","last_peered":"2024-09-11T05:03:41.629979+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:41.629979+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24738384,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9605","reported_seq":9711,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.079074+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:41.079074+0000","last_peered":"2024-09-11T05:03:41.079074+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:41.079074+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466592,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9603,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":723}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9544","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:41.418553+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:41.418553+0000","last_peered":"2024-09-11T05:03:41.418553+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:41.418553+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23467168,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9703","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:38.733393+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:38.733393+0000","last_peered":"2024-09-11T05:03:38.733393+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:38.733393+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25179296,"num_objects":3074,"num_object_clones":0,"num_object_copies":12296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9701,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":393253352,"num_objects":48010,"num_object_clones":0,"num_object_copies":192040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":786378752,"data_stored":786378752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153397,"ondisk_log_size":153397,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707012,"kb_used_data":192168,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93664828,"statfs":{"total":96636764160,"available":95912783872,"internally_reserved":0,"allocated":196780032,"data_stored":196590830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":527167184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,2,2,4,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718464,"kb_used_data":191664,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93653376,"statfs":{"total":96636764160,"available":95901057024,"internally_reserved":0,"allocated":196263936,"data_stored":196074734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621008,"kb_used_data":192320,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93750832,"statfs":{"total":96636764160,"available":96000851968,"internally_reserved":0,"allocated":196935680,"data_stored":196746478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":438955730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717220,"kb_used_data":194124,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93654620,"statfs":{"total":96636764160,"available":95902330880,"internally_reserved":0,"allocated":198782976,"data_stored":198593774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535621327},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,4,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197808128,"data_stored":197808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196386816,"data_stored":196386816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195854336,"data_stored":195854336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":196329472,"data_stored":196329472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:48.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:49.090 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:49.121 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-11T05:03:47.817001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":384979432,"num_objects":47000,"num_object_clones":0,"num_object_copies":188000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138903,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154407,"ondisk_log_size":154407,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2783604,"kb_used_data":754100,"kb_used_omap":14,"kb_used_meta":2029425,"kb_avail":374703756,"statfs":{"total":386547056640,"available":383696646144,"internally_reserved":0,"allocated":772198400,"data_stored":771441592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2078131766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,14,5,10,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22519808,"num_objects":-2749,"num_object_clones":0,"num_object_copies":-10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005034"},"pg_stats":[{"pgid":"1.f","version":"35'9574","reported_seq":9668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.993800+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:45.993800+0000","last_peered":"2024-09-11T05:03:45.993800+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:45.993800+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24084480,"num_objects":2940,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9700","reported_seq":9813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:44.470775+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:44.470775+0000","last_peered":"2024-09-11T05:03:44.470775+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:44.470775+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9700,"log_dups_size":0,"ondisk_log_size":9700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884384,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9698,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9305","reported_seq":9409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.698946+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:46.698946+0000","last_peered":"2024-09-11T05:03:46.698946+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:46.698946+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9305,"log_dups_size":0,"ondisk_log_size":9305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23386728,"num_objects":2856,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9304,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":645}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9503","reported_seq":9610,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:44.261320+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:44.261320+0000","last_peered":"2024-09-11T05:03:44.261320+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:44.261320+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9503,"log_dups_size":0,"ondisk_log_size":9503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23860384,"num_objects":2913,"num_object_clones":0,"num_object_copies":11652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9501,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9612","reported_seq":9735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.101510+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:46.101510+0000","last_peered":"2024-09-11T05:03:46.101510+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:46.101510+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23909536,"num_objects":2919,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9610,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9691","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.950063+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:45.950063+0000","last_peered":"2024-09-11T05:03:45.950063+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:45.950063+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9691,"log_dups_size":0,"ondisk_log_size":9691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24158208,"num_objects":2949,"num_object_clones":0,"num_object_copies":11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9726","reported_seq":9819,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:44.216440+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:44.216440+0000","last_peered":"2024-09-11T05:03:44.216440+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:44.216440+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24605856,"num_objects":3004,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9724,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9851","reported_seq":14708,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:46.367316+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:46.367316+0000","last_peered":"2024-09-11T05:03:46.367316+0000","last_clean":"2024-09-11T05:03:46.367316+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:46.367316+0000","last_undegraded":"2024-09-11T05:03:46.367316+0000","last_fullsized":"2024-09-11T05:03:46.367316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24346624,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9841","reported_seq":9963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.007342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:46.007342+0000","last_peered":"2024-09-11T05:03:46.007342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:46.007342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9841,"log_dups_size":0,"ondisk_log_size":9841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24136000,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":581}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9622","reported_seq":9734,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.564566+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:46.564566+0000","last_peered":"2024-09-11T05:03:46.564566+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:46.564566+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23803040,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9620,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9675","reported_seq":9791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.094950+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:46.094950+0000","last_peered":"2024-09-11T05:03:46.094950+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:46.094950+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24391408,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9489","reported_seq":14089,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:46.170731+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:46.170731+0000","last_peered":"2024-09-11T05:03:46.170731+0000","last_clean":"2024-09-11T05:03:46.170731+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:46.170731+0000","last_undegraded":"2024-09-11T05:03:46.170731+0000","last_fullsized":"2024-09-11T05:03:46.170731+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9489,"log_dups_size":0,"ondisk_log_size":9489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23715840,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3299,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9780","reported_seq":10859,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.725568+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:46.725568+0000","last_peered":"2024-09-11T05:03:46.725568+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:46.725568+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9780,"log_dups_size":0,"ondisk_log_size":9780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24107600,"num_objects":2943,"num_object_clones":0,"num_object_copies":11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9779,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9668","reported_seq":9774,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.913267+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:45.913267+0000","last_peered":"2024-09-11T05:03:45.913267+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:45.913267+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23950496,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9596","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.645823+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:46.645823+0000","last_peered":"2024-09-11T05:03:46.645823+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:46.645823+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9596,"log_dups_size":0,"ondisk_log_size":9596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041184,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9594,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9774","reported_seq":9893,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:44.196092+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:44.196092+0000","last_peered":"2024-09-11T05:03:44.196092+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:44.196092+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24597664,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":384979432,"num_objects":47000,"num_object_clones":0,"num_object_copies":188000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138903,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":769679360,"data_stored":769679360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154407,"ondisk_log_size":154407,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725136,"kb_used_data":188096,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93646704,"statfs":{"total":96636764160,"available":95894224896,"internally_reserved":0,"allocated":192610304,"data_stored":192421102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":549908176},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720036,"kb_used_data":187668,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93651804,"statfs":{"total":96636764160,"available":95899447296,"internally_reserved":0,"allocated":192172032,"data_stored":191982830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621108,"kb_used_data":188260,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93750732,"statfs":{"total":96636764160,"available":96000749568,"internally_reserved":0,"allocated":192778240,"data_stored":192589038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443215570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717324,"kb_used_data":190076,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93654516,"statfs":{"total":96636764160,"available":95902224384,"internally_reserved":0,"allocated":194637824,"data_stored":194448622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539881167},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,1,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193630208,"data_stored":193630208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192118784,"data_stored":192118784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":191725568,"data_stored":191725568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192204800,"data_stored":192204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:49.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:49.772 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:49.773 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:49.800 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-11T05:03:47.817001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":384979432,"num_objects":47000,"num_object_clones":0,"num_object_copies":188000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138903,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154407,"ondisk_log_size":154407,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2783604,"kb_used_data":754100,"kb_used_omap":14,"kb_used_meta":2029425,"kb_avail":374703756,"statfs":{"total":386547056640,"available":383696646144,"internally_reserved":0,"allocated":772198400,"data_stored":771441592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2078131766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,14,5,10,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22519808,"num_objects":-2749,"num_object_clones":0,"num_object_copies":-10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005034"},"pg_stats":[{"pgid":"1.f","version":"35'9574","reported_seq":9668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.993800+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:45.993800+0000","last_peered":"2024-09-11T05:03:45.993800+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:45.993800+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24084480,"num_objects":2940,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9700","reported_seq":9813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:44.470775+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:44.470775+0000","last_peered":"2024-09-11T05:03:44.470775+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:44.470775+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9700,"log_dups_size":0,"ondisk_log_size":9700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884384,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9698,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9305","reported_seq":9409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.698946+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:46.698946+0000","last_peered":"2024-09-11T05:03:46.698946+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:46.698946+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9305,"log_dups_size":0,"ondisk_log_size":9305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23386728,"num_objects":2856,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9304,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":645}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9503","reported_seq":9610,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:44.261320+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:44.261320+0000","last_peered":"2024-09-11T05:03:44.261320+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:44.261320+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9503,"log_dups_size":0,"ondisk_log_size":9503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23860384,"num_objects":2913,"num_object_clones":0,"num_object_copies":11652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9501,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9612","reported_seq":9735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.101510+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:46.101510+0000","last_peered":"2024-09-11T05:03:46.101510+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:46.101510+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23909536,"num_objects":2919,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9610,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9691","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.950063+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:45.950063+0000","last_peered":"2024-09-11T05:03:45.950063+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:45.950063+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9691,"log_dups_size":0,"ondisk_log_size":9691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24158208,"num_objects":2949,"num_object_clones":0,"num_object_copies":11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9726","reported_seq":9819,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:44.216440+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:44.216440+0000","last_peered":"2024-09-11T05:03:44.216440+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:44.216440+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24605856,"num_objects":3004,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9724,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9851","reported_seq":14708,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:46.367316+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:46.367316+0000","last_peered":"2024-09-11T05:03:46.367316+0000","last_clean":"2024-09-11T05:03:46.367316+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:46.367316+0000","last_undegraded":"2024-09-11T05:03:46.367316+0000","last_fullsized":"2024-09-11T05:03:46.367316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24346624,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9841","reported_seq":9963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.007342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:46.007342+0000","last_peered":"2024-09-11T05:03:46.007342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:46.007342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9841,"log_dups_size":0,"ondisk_log_size":9841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24136000,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":581}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9622","reported_seq":9734,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.564566+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:46.564566+0000","last_peered":"2024-09-11T05:03:46.564566+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:46.564566+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23803040,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9620,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9675","reported_seq":9791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.094950+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:46.094950+0000","last_peered":"2024-09-11T05:03:46.094950+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:46.094950+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24391408,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9489","reported_seq":14089,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:46.170731+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:46.170731+0000","last_peered":"2024-09-11T05:03:46.170731+0000","last_clean":"2024-09-11T05:03:46.170731+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:46.170731+0000","last_undegraded":"2024-09-11T05:03:46.170731+0000","last_fullsized":"2024-09-11T05:03:46.170731+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9489,"log_dups_size":0,"ondisk_log_size":9489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23715840,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3299,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9780","reported_seq":10859,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.725568+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:46.725568+0000","last_peered":"2024-09-11T05:03:46.725568+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:46.725568+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9780,"log_dups_size":0,"ondisk_log_size":9780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24107600,"num_objects":2943,"num_object_clones":0,"num_object_copies":11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9779,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9668","reported_seq":9774,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.913267+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:45.913267+0000","last_peered":"2024-09-11T05:03:45.913267+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:45.913267+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23950496,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9596","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.645823+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:46.645823+0000","last_peered":"2024-09-11T05:03:46.645823+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:46.645823+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9596,"log_dups_size":0,"ondisk_log_size":9596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041184,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9594,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9774","reported_seq":9893,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:44.196092+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:44.196092+0000","last_peered":"2024-09-11T05:03:44.196092+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:44.196092+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24597664,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":384979432,"num_objects":47000,"num_object_clones":0,"num_object_copies":188000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138903,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":769679360,"data_stored":769679360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154407,"ondisk_log_size":154407,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725136,"kb_used_data":188096,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93646704,"statfs":{"total":96636764160,"available":95894224896,"internally_reserved":0,"allocated":192610304,"data_stored":192421102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":549908176},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720036,"kb_used_data":187668,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93651804,"statfs":{"total":96636764160,"available":95899447296,"internally_reserved":0,"allocated":192172032,"data_stored":191982830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621108,"kb_used_data":188260,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93750732,"statfs":{"total":96636764160,"available":96000749568,"internally_reserved":0,"allocated":192778240,"data_stored":192589038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443215570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717324,"kb_used_data":190076,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93654516,"statfs":{"total":96636764160,"available":95902224384,"internally_reserved":0,"allocated":194637824,"data_stored":194448622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539881167},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,1,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193630208,"data_stored":193630208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192118784,"data_stored":192118784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":191725568,"data_stored":191725568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192204800,"data_stored":192204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:49.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:49.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9574", 'reported_seq': 9668, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:45.993800+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:45.993800+0000', 'last_peered': '2024-09-11T05:03:45.993800+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:45.993800+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9574, 'log_dups_size': 0, 'ondisk_log_size': 9574, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24084480, 'num_objects': 2940, 'num_object_clones': 0, 'num_object_copies': 11760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9574, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 649}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:49.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:49.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9700", 'reported_seq': 9813, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:44.470775+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:44.470775+0000', 'last_peered': '2024-09-11T05:03:44.470775+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:44.470775+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9700, 'log_dups_size': 0, 'ondisk_log_size': 9700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24884384, 'num_objects': 3038, 'num_object_clones': 0, 'num_object_copies': 12152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9698, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 780}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:49.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:49.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9305", 'reported_seq': 9409, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:46.698946+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:46.698946+0000', 'last_peered': '2024-09-11T05:03:46.698946+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:46.698946+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9305, 'log_dups_size': 0, 'ondisk_log_size': 9305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23386728, 'num_objects': 2856, 'num_object_clones': 0, 'num_object_copies': 11424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2856, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9304, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 645}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:49.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:49.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9503", 'reported_seq': 9610, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:44.261320+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:44.261320+0000', 'last_peered': '2024-09-11T05:03:44.261320+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:44.261320+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9503, 'log_dups_size': 0, 'ondisk_log_size': 9503, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23860384, 'num_objects': 2913, 'num_object_clones': 0, 'num_object_copies': 11652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9501, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9612", 'reported_seq': 9735, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:46.101510+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:46.101510+0000', 'last_peered': '2024-09-11T05:03:46.101510+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:46.101510+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9612, 'log_dups_size': 0, 'ondisk_log_size': 9612, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23909536, 'num_objects': 2919, 'num_object_clones': 0, 'num_object_copies': 11676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9610, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 698}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9691", 'reported_seq': 9796, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:45.950063+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:45.950063+0000', 'last_peered': '2024-09-11T05:03:45.950063+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:45.950063+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9691, 'log_dups_size': 0, 'ondisk_log_size': 9691, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24158208, 'num_objects': 2949, 'num_object_clones': 0, 'num_object_copies': 11796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9691, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 655}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9726", 'reported_seq': 9819, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:44.216440+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:44.216440+0000', 'last_peered': '2024-09-11T05:03:44.216440+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:44.216440+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9726, 'log_dups_size': 0, 'ondisk_log_size': 9726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24605856, 'num_objects': 3004, 'num_object_clones': 0, 'num_object_copies': 12016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9724, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 697}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9841", 'reported_seq': 9963, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:46.007342+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:46.007342+0000', 'last_peered': '2024-09-11T05:03:46.007342+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:46.007342+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9841, 'log_dups_size': 0, 'ondisk_log_size': 9841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24136000, 'num_objects': 2947, 'num_object_clones': 0, 'num_object_copies': 11788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9838, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 581}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9622", 'reported_seq': 9734, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:46.564566+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:46.564566+0000', 'last_peered': '2024-09-11T05:03:46.564566+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:46.564566+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9622, 'log_dups_size': 0, 'ondisk_log_size': 9622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23803040, 'num_objects': 2906, 'num_object_clones': 0, 'num_object_copies': 11624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9620, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 600}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:49.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9675", 'reported_seq': 9791, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:46.094950+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:46.094950+0000', 'last_peered': '2024-09-11T05:03:46.094950+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:46.094950+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9675, 'log_dups_size': 0, 'ondisk_log_size': 9675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24391408, 'num_objects': 2978, 'num_object_clones': 0, 'num_object_copies': 11912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9672, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 680}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:49.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:49.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9780", 'reported_seq': 10859, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:46.725568+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:46.725568+0000', 'last_peered': '2024-09-11T05:03:46.725568+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:46.725568+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9780, 'log_dups_size': 0, 'ondisk_log_size': 9780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24107600, 'num_objects': 2943, 'num_object_clones': 0, 'num_object_copies': 11772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1834, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9779, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 500, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:49.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:49.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9668", 'reported_seq': 9774, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:45.913267+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:45.913267+0000', 'last_peered': '2024-09-11T05:03:45.913267+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:45.913267+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9668, 'log_dups_size': 0, 'ondisk_log_size': 9668, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23950496, 'num_objects': 2924, 'num_object_clones': 0, 'num_object_copies': 11696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9666, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 660}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:49.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:49.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9596", 'reported_seq': 9714, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:46.645823+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:46.645823+0000', 'last_peered': '2024-09-11T05:03:46.645823+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:46.645823+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9596, 'log_dups_size': 0, 'ondisk_log_size': 9596, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23041184, 'num_objects': 2813, 'num_object_clones': 0, 'num_object_copies': 11252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9594, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:49.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:49.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9774", 'reported_seq': 9893, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:44.196092+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:44.196092+0000', 'last_peered': '2024-09-11T05:03:44.196092+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:44.196092+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9774, 'log_dups_size': 0, 'ondisk_log_size': 9774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24597664, 'num_objects': 3003, 'num_object_clones': 0, 'num_object_copies': 12012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9772, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 715}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:49.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:50.589 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:50.589 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31525/186932 objects degraded (16.864%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":557,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":46733,"data_bytes":382792168,"bytes_used":2851164160,"bytes_avail":383695892480,"bytes_total":386547056640,"degraded_objects":31525,"degraded_total":186932,"degraded_ratio":0.16864421286885067,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":187},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:50.609 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:03:50.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:51.290 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:51.291 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:51.309 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-11T05:03:49.817708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":382792168,"num_objects":46733,"num_object_clones":0,"num_object_copies":186932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139170,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154674,"ondisk_log_size":154674,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784340,"kb_used_data":749268,"kb_used_omap":14,"kb_used_meta":2034993,"kb_avail":374703020,"statfs":{"total":386547056640,"available":383695892480,"internally_reserved":0,"allocated":767250432,"data_stored":766493624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2083833398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,6,12,7,11,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18432000,"num_objects":-2250,"num_object_clones":0,"num_object_copies":-9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005027"},"pg_stats":[{"pgid":"1.f","version":"35'9574","reported_seq":9668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.993800+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:45.993800+0000","last_peered":"2024-09-11T05:03:45.993800+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:45.993800+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24084480,"num_objects":2940,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9765","reported_seq":9879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.331692+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:49.331692+0000","last_peered":"2024-09-11T05:03:49.331692+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:49.331692+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24351904,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9305","reported_seq":9409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.698946+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:46.698946+0000","last_peered":"2024-09-11T05:03:46.698946+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:46.698946+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9305,"log_dups_size":0,"ondisk_log_size":9305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23386728,"num_objects":2856,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9304,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":645}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9574","reported_seq":9682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.467679+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:49.467679+0000","last_peered":"2024-09-11T05:03:49.467679+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:49.467679+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23278752,"num_objects":2842,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9612","reported_seq":9735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.101510+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:46.101510+0000","last_peered":"2024-09-11T05:03:46.101510+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:46.101510+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23909536,"num_objects":2919,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9610,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9691","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.950063+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:45.950063+0000","last_peered":"2024-09-11T05:03:45.950063+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:45.950063+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9691,"log_dups_size":0,"ondisk_log_size":9691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24158208,"num_objects":2949,"num_object_clones":0,"num_object_copies":11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9798","reported_seq":9892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.535336+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:49.535336+0000","last_peered":"2024-09-11T05:03:49.535336+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:49.535336+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9798,"log_dups_size":0,"ondisk_log_size":9798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016032,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9796,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":625}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9851","reported_seq":14708,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:46.367316+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:46.367316+0000","last_peered":"2024-09-11T05:03:46.367316+0000","last_clean":"2024-09-11T05:03:46.367316+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:46.367316+0000","last_undegraded":"2024-09-11T05:03:46.367316+0000","last_fullsized":"2024-09-11T05:03:46.367316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24346624,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9841","reported_seq":9963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.007342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:46.007342+0000","last_peered":"2024-09-11T05:03:46.007342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:46.007342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9841,"log_dups_size":0,"ondisk_log_size":9841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24136000,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":581}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9622","reported_seq":9734,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.564566+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:46.564566+0000","last_peered":"2024-09-11T05:03:46.564566+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:46.564566+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23803040,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9620,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9675","reported_seq":9791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.094950+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:46.094950+0000","last_peered":"2024-09-11T05:03:46.094950+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:46.094950+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24391408,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9489","reported_seq":14089,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:46.170731+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:46.170731+0000","last_peered":"2024-09-11T05:03:46.170731+0000","last_clean":"2024-09-11T05:03:46.170731+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:46.170731+0000","last_undegraded":"2024-09-11T05:03:46.170731+0000","last_fullsized":"2024-09-11T05:03:46.170731+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9489,"log_dups_size":0,"ondisk_log_size":9489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23715840,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3299,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9780","reported_seq":10859,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.725568+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:46.725568+0000","last_peered":"2024-09-11T05:03:46.725568+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:46.725568+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9780,"log_dups_size":0,"ondisk_log_size":9780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24107600,"num_objects":2943,"num_object_clones":0,"num_object_copies":11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9779,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9668","reported_seq":9774,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:45.913267+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:45.913267+0000","last_peered":"2024-09-11T05:03:45.913267+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:45.913267+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23950496,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9596","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:46.645823+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:46.645823+0000","last_peered":"2024-09-11T05:03:46.645823+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:46.645823+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9596,"log_dups_size":0,"ondisk_log_size":9596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041184,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9594,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9833","reported_seq":9952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.160001+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:49.160001+0000","last_peered":"2024-09-11T05:03:49.160001+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:49.160001+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9831,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":656}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":382792168,"num_objects":46733,"num_object_clones":0,"num_object_copies":186932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139170,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":765431808,"data_stored":765431808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154674,"ondisk_log_size":154674,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725136,"kb_used_data":188096,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93646704,"statfs":{"total":96636764160,"available":95894224896,"internally_reserved":0,"allocated":192610304,"data_stored":192421102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":549908176},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720036,"kb_used_data":187668,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93651804,"statfs":{"total":96636764160,"available":95899447296,"internally_reserved":0,"allocated":192172032,"data_stored":191982830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621108,"kb_used_data":188260,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93750732,"statfs":{"total":96636764160,"available":96000749568,"internally_reserved":0,"allocated":192778240,"data_stored":192589038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443215570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718060,"kb_used_data":185244,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93653780,"statfs":{"total":96636764160,"available":95901470720,"internally_reserved":0,"allocated":189689856,"data_stored":189500654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189382656,"data_stored":189382656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192118784,"data_stored":192118784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":191725568,"data_stored":191725568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192204800,"data_stored":192204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:54.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:54.934 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:54.955 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-11T05:03:53.819232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376574440,"num_objects":45974,"num_object_clones":0,"num_object_copies":183896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139929,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155433,"ondisk_log_size":155433,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784856,"kb_used_data":737276,"kb_used_omap":14,"kb_used_meta":2047473,"kb_avail":374702504,"statfs":{"total":386547056640,"available":383695364096,"internally_reserved":0,"allocated":754970624,"data_stored":754213816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2096612918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,9,6,12,15,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16678912,"num_objects":-2036,"num_object_clones":0,"num_object_copies":-8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004827"},"pg_stats":[{"pgid":"1.f","version":"35'9627","reported_seq":9721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.096719+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:51.096719+0000","last_peered":"2024-09-11T05:03:51.096719+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:51.096719+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":2887,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9627,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9765","reported_seq":9879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.331692+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:49.331692+0000","last_peered":"2024-09-11T05:03:49.331692+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:49.331692+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24351904,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9364","reported_seq":9468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.784756+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:51.784756+0000","last_peered":"2024-09-11T05:03:51.784756+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:51.784756+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22903400,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9363,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":586}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9574","reported_seq":9682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.467679+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:49.467679+0000","last_peered":"2024-09-11T05:03:49.467679+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:49.467679+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23278752,"num_objects":2842,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9676","reported_seq":9799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.177571+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:51.177571+0000","last_peered":"2024-09-11T05:03:51.177571+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:51.177571+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9676,"log_dups_size":0,"ondisk_log_size":9676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23385248,"num_objects":2855,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9674,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9761","reported_seq":9866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.237973+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:51.237973+0000","last_peered":"2024-09-11T05:03:51.237973+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:51.237973+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23584768,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9798","reported_seq":9892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.535336+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:49.535336+0000","last_peered":"2024-09-11T05:03:49.535336+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:49.535336+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9798,"log_dups_size":0,"ondisk_log_size":9798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016032,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9796,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":625}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9916","reported_seq":14773,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:51.256524+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:51.256524+0000","last_peered":"2024-09-11T05:03:51.256524+0000","last_clean":"2024-09-11T05:03:51.256524+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:51.256524+0000","last_undegraded":"2024-09-11T05:03:51.256524+0000","last_fullsized":"2024-09-11T05:03:51.256524+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9916,"log_dups_size":0,"ondisk_log_size":9916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23814144,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":626,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9904","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:50.985132+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:50.985132+0000","last_peered":"2024-09-11T05:03:50.985132+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:50.985132+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23619904,"num_objects":2884,"num_object_clones":0,"num_object_copies":11536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":518}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9684","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.793141+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:51.793141+0000","last_peered":"2024-09-11T05:03:51.793141+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:51.793141+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23295136,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9682,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":538}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9741","reported_seq":9857,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.265951+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:51.265951+0000","last_peered":"2024-09-11T05:03:51.265951+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:51.265951+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9741,"log_dups_size":0,"ondisk_log_size":9741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23850736,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9738,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9558","reported_seq":14158,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:51.002903+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:51.002903+0000","last_peered":"2024-09-11T05:03:51.002903+0000","last_clean":"2024-09-11T05:03:51.002903+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:51.002903+0000","last_undegraded":"2024-09-11T05:03:51.002903+0000","last_fullsized":"2024-09-11T05:03:51.002903+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23150592,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9850","reported_seq":11023,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.732571+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:51.732571+0000","last_peered":"2024-09-11T05:03:51.732571+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:51.732571+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23534160,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9849,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1085}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9726","reported_seq":9832,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.172214+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:51.172214+0000","last_peered":"2024-09-11T05:03:51.172214+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:51.172214+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23475360,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9724,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":602}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9656","reported_seq":9774,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.629813+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:51.629813+0000","last_peered":"2024-09-11T05:03:51.629813+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:51.629813+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9656,"log_dups_size":0,"ondisk_log_size":9656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22549664,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9833","reported_seq":9952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.160001+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:49.160001+0000","last_peered":"2024-09-11T05:03:49.160001+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:49.160001+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9831,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":656}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376574440,"num_objects":45974,"num_object_clones":0,"num_object_copies":183896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139929,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":753008640,"data_stored":753008640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155433,"ondisk_log_size":155433,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725272,"kb_used_data":184072,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93646568,"statfs":{"total":96636764160,"available":95894085632,"internally_reserved":0,"allocated":188489728,"data_stored":188300526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":554168016},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720204,"kb_used_data":183672,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93651636,"statfs":{"total":96636764160,"available":95899275264,"internally_reserved":0,"allocated":188080128,"data_stored":187890926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,3,0,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621320,"kb_used_data":184288,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93750520,"statfs":{"total":96636764160,"available":96000532480,"internally_reserved":0,"allocated":188710912,"data_stored":188521710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447475410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718060,"kb_used_data":185244,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93653780,"statfs":{"total":96636764160,"available":95901470720,"internally_reserved":0,"allocated":189689856,"data_stored":189500654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189382656,"data_stored":189382656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188022784,"data_stored":188022784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187551744,"data_stored":187551744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":188051456,"data_stored":188051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:54.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:55.632 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:55.662 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-11T05:03:53.819232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376574440,"num_objects":45974,"num_object_clones":0,"num_object_copies":183896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139929,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155433,"ondisk_log_size":155433,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784856,"kb_used_data":737276,"kb_used_omap":14,"kb_used_meta":2047473,"kb_avail":374702504,"statfs":{"total":386547056640,"available":383695364096,"internally_reserved":0,"allocated":754970624,"data_stored":754213816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2096612918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,9,6,12,15,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16678912,"num_objects":-2036,"num_object_clones":0,"num_object_copies":-8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004827"},"pg_stats":[{"pgid":"1.f","version":"35'9627","reported_seq":9721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.096719+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:51.096719+0000","last_peered":"2024-09-11T05:03:51.096719+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:51.096719+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":2887,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9627,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9765","reported_seq":9879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.331692+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:49.331692+0000","last_peered":"2024-09-11T05:03:49.331692+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:49.331692+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24351904,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9364","reported_seq":9468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.784756+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:51.784756+0000","last_peered":"2024-09-11T05:03:51.784756+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:51.784756+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22903400,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9363,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":586}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9574","reported_seq":9682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.467679+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:49.467679+0000","last_peered":"2024-09-11T05:03:49.467679+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:49.467679+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23278752,"num_objects":2842,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9676","reported_seq":9799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.177571+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:51.177571+0000","last_peered":"2024-09-11T05:03:51.177571+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:51.177571+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9676,"log_dups_size":0,"ondisk_log_size":9676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23385248,"num_objects":2855,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9674,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9761","reported_seq":9866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.237973+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:51.237973+0000","last_peered":"2024-09-11T05:03:51.237973+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:51.237973+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23584768,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9798","reported_seq":9892,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.535336+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:49.535336+0000","last_peered":"2024-09-11T05:03:49.535336+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:49.535336+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9798,"log_dups_size":0,"ondisk_log_size":9798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016032,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9796,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":625}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9916","reported_seq":14773,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:51.256524+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:51.256524+0000","last_peered":"2024-09-11T05:03:51.256524+0000","last_clean":"2024-09-11T05:03:51.256524+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:51.256524+0000","last_undegraded":"2024-09-11T05:03:51.256524+0000","last_fullsized":"2024-09-11T05:03:51.256524+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9916,"log_dups_size":0,"ondisk_log_size":9916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23814144,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":626,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9904","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:50.985132+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:50.985132+0000","last_peered":"2024-09-11T05:03:50.985132+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:50.985132+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23619904,"num_objects":2884,"num_object_clones":0,"num_object_copies":11536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":518}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9684","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.793141+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:51.793141+0000","last_peered":"2024-09-11T05:03:51.793141+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:51.793141+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23295136,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9682,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":538}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9741","reported_seq":9857,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.265951+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:51.265951+0000","last_peered":"2024-09-11T05:03:51.265951+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:51.265951+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9741,"log_dups_size":0,"ondisk_log_size":9741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23850736,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9738,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9558","reported_seq":14158,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:51.002903+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:51.002903+0000","last_peered":"2024-09-11T05:03:51.002903+0000","last_clean":"2024-09-11T05:03:51.002903+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:51.002903+0000","last_undegraded":"2024-09-11T05:03:51.002903+0000","last_fullsized":"2024-09-11T05:03:51.002903+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23150592,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9850","reported_seq":11023,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.732571+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:51.732571+0000","last_peered":"2024-09-11T05:03:51.732571+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:51.732571+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23534160,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9849,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1085}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9726","reported_seq":9832,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.172214+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:51.172214+0000","last_peered":"2024-09-11T05:03:51.172214+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:51.172214+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23475360,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9724,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":602}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9656","reported_seq":9774,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.629813+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:51.629813+0000","last_peered":"2024-09-11T05:03:51.629813+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:51.629813+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9656,"log_dups_size":0,"ondisk_log_size":9656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22549664,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9833","reported_seq":9952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:49.160001+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:49.160001+0000","last_peered":"2024-09-11T05:03:49.160001+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:49.160001+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114336,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9831,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":656}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376574440,"num_objects":45974,"num_object_clones":0,"num_object_copies":183896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139929,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":753008640,"data_stored":753008640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155433,"ondisk_log_size":155433,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725272,"kb_used_data":184072,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93646568,"statfs":{"total":96636764160,"available":95894085632,"internally_reserved":0,"allocated":188489728,"data_stored":188300526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":554168016},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720204,"kb_used_data":183672,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93651636,"statfs":{"total":96636764160,"available":95899275264,"internally_reserved":0,"allocated":188080128,"data_stored":187890926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,3,0,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621320,"kb_used_data":184288,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93750520,"statfs":{"total":96636764160,"available":96000532480,"internally_reserved":0,"allocated":188710912,"data_stored":188521710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447475410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718060,"kb_used_data":185244,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93653780,"statfs":{"total":96636764160,"available":95901470720,"internally_reserved":0,"allocated":189689856,"data_stored":189500654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189382656,"data_stored":189382656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188022784,"data_stored":188022784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187551744,"data_stored":187551744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":188051456,"data_stored":188051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:03:55.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:03:55.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9627", 'reported_seq': 9721, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.096719+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:51.096719+0000', 'last_peered': '2024-09-11T05:03:51.096719+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:51.096719+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9627, 'log_dups_size': 0, 'ondisk_log_size': 9627, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23650304, 'num_objects': 2887, 'num_object_clones': 0, 'num_object_copies': 11548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9627, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 596}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:55.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:03:55.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9765", 'reported_seq': 9879, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:49.331692+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:49.331692+0000', 'last_peered': '2024-09-11T05:03:49.331692+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:49.331692+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9765, 'log_dups_size': 0, 'ondisk_log_size': 9765, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24351904, 'num_objects': 2973, 'num_object_clones': 0, 'num_object_copies': 11892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9763, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 715}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:55.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:03:55.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9364", 'reported_seq': 9468, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.784756+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:51.784756+0000', 'last_peered': '2024-09-11T05:03:51.784756+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:51.784756+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9364, 'log_dups_size': 0, 'ondisk_log_size': 9364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22903400, 'num_objects': 2797, 'num_object_clones': 0, 'num_object_copies': 11188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2797, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9363, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 586}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:55.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:03:55.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9574", 'reported_seq': 9682, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:49.467679+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:49.467679+0000', 'last_peered': '2024-09-11T05:03:49.467679+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:49.467679+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9574, 'log_dups_size': 0, 'ondisk_log_size': 9574, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23278752, 'num_objects': 2842, 'num_object_clones': 0, 'num_object_copies': 11368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9572, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 581}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:55.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:03:55.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9676", 'reported_seq': 9799, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.177571+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:51.177571+0000', 'last_peered': '2024-09-11T05:03:51.177571+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:51.177571+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9676, 'log_dups_size': 0, 'ondisk_log_size': 9676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23385248, 'num_objects': 2855, 'num_object_clones': 0, 'num_object_copies': 11420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9674, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 634}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:55.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:03:55.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9761", 'reported_seq': 9866, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.237973+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:51.237973+0000', 'last_peered': '2024-09-11T05:03:51.237973+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:51.237973+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9761, 'log_dups_size': 0, 'ondisk_log_size': 9761, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23584768, 'num_objects': 2879, 'num_object_clones': 0, 'num_object_copies': 11516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9761, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 585}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:55.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:03:55.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9798", 'reported_seq': 9892, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:49.535336+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:49.535336+0000', 'last_peered': '2024-09-11T05:03:49.535336+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:49.535336+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9798, 'log_dups_size': 0, 'ondisk_log_size': 9798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24016032, 'num_objects': 2932, 'num_object_clones': 0, 'num_object_copies': 11728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9796, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 625}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9904", 'reported_seq': 10026, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:50.985132+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:50.985132+0000', 'last_peered': '2024-09-11T05:03:50.985132+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:50.985132+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9904, 'log_dups_size': 0, 'ondisk_log_size': 9904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23619904, 'num_objects': 2884, 'num_object_clones': 0, 'num_object_copies': 11536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9901, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 518}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9684", 'reported_seq': 9796, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.793141+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:51.793141+0000', 'last_peered': '2024-09-11T05:03:51.793141+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:51.793141+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9684, 'log_dups_size': 0, 'ondisk_log_size': 9684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23295136, 'num_objects': 2844, 'num_object_clones': 0, 'num_object_copies': 11376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9682, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 538}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9741", 'reported_seq': 9857, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.265951+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:51.265951+0000', 'last_peered': '2024-09-11T05:03:51.265951+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:51.265951+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9741, 'log_dups_size': 0, 'ondisk_log_size': 9741, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23850736, 'num_objects': 2912, 'num_object_clones': 0, 'num_object_copies': 11648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9738, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 614}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9850", 'reported_seq': 11023, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.732571+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:51.732571+0000', 'last_peered': '2024-09-11T05:03:51.732571+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:51.732571+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9850, 'log_dups_size': 0, 'ondisk_log_size': 9850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23534160, 'num_objects': 2873, 'num_object_clones': 0, 'num_object_copies': 11492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1788, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9849, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 546, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1085}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:03:55.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9726", 'reported_seq': 9832, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.172214+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:51.172214+0000', 'last_peered': '2024-09-11T05:03:51.172214+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:51.172214+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9726, 'log_dups_size': 0, 'ondisk_log_size': 9726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23475360, 'num_objects': 2866, 'num_object_clones': 0, 'num_object_copies': 11464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9724, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 602}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:03:55.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:03:55.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9656", 'reported_seq': 9774, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:51.629813+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:51.629813+0000', 'last_peered': '2024-09-11T05:03:51.629813+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:51.629813+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9656, 'log_dups_size': 0, 'ondisk_log_size': 9656, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22549664, 'num_objects': 2753, 'num_object_clones': 0, 'num_object_copies': 11012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9654, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:03:55.666 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:03:55.666 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9833", 'reported_seq': 9952, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:49.160001+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:49.160001+0000', 'last_peered': '2024-09-11T05:03:49.160001+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:49.160001+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9833, 'log_dups_size': 0, 'ondisk_log_size': 9833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24114336, 'num_objects': 2944, 'num_object_clones': 0, 'num_object_copies': 11776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9831, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 656}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:03:55.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:03:56.434 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:56.434 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31479/182908 objects degraded (17.210%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":563,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45727,"data_bytes":374551016,"bytes_used":2870329344,"bytes_avail":383676727296,"bytes_total":386547056640,"degraded_objects":31479,"degraded_total":182908,"degraded_ratio":0.17210291512673037,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":190},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:03:56.458 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:03:56.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:03:57.096 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:03:57.097 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:03:57.113 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-11T05:03:55.820046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":374551016,"num_objects":45727,"num_object_clones":0,"num_object_copies":182908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45727,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140176,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155680,"ondisk_log_size":155680,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803056,"kb_used_data":733264,"kb_used_omap":14,"kb_used_meta":2069681,"kb_avail":374684304,"statfs":{"total":386547056640,"available":383676727296,"internally_reserved":0,"allocated":750862336,"data_stored":750105528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2119353910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,5,10,14,18,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18702336,"num_objects":-2283,"num_object_clones":0,"num_object_copies":-9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2283,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004745"},"pg_stats":[{"pgid":"1.f","version":"35'9627","reported_seq":9721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.096719+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:51.096719+0000","last_peered":"2024-09-11T05:03:51.096719+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:51.096719+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":2887,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9627,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9844","reported_seq":9956,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:54.495133+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:54.495133+0000","last_peered":"2024-09-11T05:03:54.495133+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:54.495133+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9844,"log_dups_size":0,"ondisk_log_size":9844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23704736,"num_objects":2894,"num_object_clones":0,"num_object_copies":11576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9842,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9364","reported_seq":9468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.784756+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:51.784756+0000","last_peered":"2024-09-11T05:03:51.784756+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:51.784756+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22903400,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9363,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":586}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9629","reported_seq":9737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:54.467452+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:54.467452+0000","last_peered":"2024-09-11T05:03:54.467452+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:54.467452+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22828192,"num_objects":2787,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9627,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9676","reported_seq":9799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.177571+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:51.177571+0000","last_peered":"2024-09-11T05:03:51.177571+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:51.177571+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9676,"log_dups_size":0,"ondisk_log_size":9676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23385248,"num_objects":2855,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9674,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9761","reported_seq":9866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.237973+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:51.237973+0000","last_peered":"2024-09-11T05:03:51.237973+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:51.237973+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23584768,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9855","reported_seq":9949,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:54.480203+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:54.480203+0000","last_peered":"2024-09-11T05:03:54.480203+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:54.480203+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9855,"log_dups_size":0,"ondisk_log_size":9855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23549088,"num_objects":2875,"num_object_clones":0,"num_object_copies":11500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9853,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":568}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9916","reported_seq":14773,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:51.256524+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:51.256524+0000","last_peered":"2024-09-11T05:03:51.256524+0000","last_clean":"2024-09-11T05:03:51.256524+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:51.256524+0000","last_undegraded":"2024-09-11T05:03:51.256524+0000","last_fullsized":"2024-09-11T05:03:51.256524+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9916,"log_dups_size":0,"ondisk_log_size":9916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23814144,"num_objects":2907,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":626,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9904","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:50.985132+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:50.985132+0000","last_peered":"2024-09-11T05:03:50.985132+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:50.985132+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23619904,"num_objects":2884,"num_object_clones":0,"num_object_copies":11536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":518}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9684","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.793141+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:51.793141+0000","last_peered":"2024-09-11T05:03:51.793141+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:51.793141+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23295136,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9682,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":538}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9741","reported_seq":9857,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.265951+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:51.265951+0000","last_peered":"2024-09-11T05:03:51.265951+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:51.265951+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9741,"log_dups_size":0,"ondisk_log_size":9741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23850736,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9738,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9558","reported_seq":14158,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:51.002903+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:51.002903+0000","last_peered":"2024-09-11T05:03:51.002903+0000","last_clean":"2024-09-11T05:03:51.002903+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:51.002903+0000","last_undegraded":"2024-09-11T05:03:51.002903+0000","last_fullsized":"2024-09-11T05:03:51.002903+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23150592,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9850","reported_seq":11023,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.732571+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:51.732571+0000","last_peered":"2024-09-11T05:03:51.732571+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:51.732571+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23534160,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9849,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1085}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9726","reported_seq":9832,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.172214+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:51.172214+0000","last_peered":"2024-09-11T05:03:51.172214+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:51.172214+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23475360,"num_objects":2866,"num_object_clones":0,"num_object_copies":11464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9724,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":602}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9656","reported_seq":9774,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:51.629813+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:51.629813+0000","last_peered":"2024-09-11T05:03:51.629813+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:51.629813+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9656,"log_dups_size":0,"ondisk_log_size":9656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22549664,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9889","reported_seq":10008,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:54.191053+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:54.191053+0000","last_peered":"2024-09-11T05:03:54.191053+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:54.191053+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23655584,"num_objects":2888,"num_object_clones":0,"num_object_copies":11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9887,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":600}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":374551016,"num_objects":45727,"num_object_clones":0,"num_object_copies":182908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45727,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140176,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":748863488,"data_stored":748863488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155680,"ondisk_log_size":155680,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725272,"kb_used_data":184072,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93646568,"statfs":{"total":96636764160,"available":95894085632,"internally_reserved":0,"allocated":188489728,"data_stored":188300526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":554168016},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720204,"kb_used_data":183672,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93651636,"statfs":{"total":96636764160,"available":95899275264,"internally_reserved":0,"allocated":188080128,"data_stored":187890926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,3,0,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621320,"kb_used_data":184288,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93750520,"statfs":{"total":96636764160,"available":96000532480,"internally_reserved":0,"allocated":188710912,"data_stored":188521710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447475410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,5,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736260,"kb_used_data":181232,"kb_used_omap":4,"kb_used_meta":555003,"kb_avail":93635580,"statfs":{"total":96636764160,"available":95882833920,"internally_reserved":0,"allocated":185581568,"data_stored":185392366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568323791},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,5,4,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185237504,"data_stored":185237504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188022784,"data_stored":188022784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187551744,"data_stored":187551744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":188051456,"data_stored":188051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:00.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:00.800 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:00.817 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-11T05:03:59.821745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":366195176,"num_objects":44707,"num_object_clones":0,"num_object_copies":178828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141196,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156700,"ondisk_log_size":156700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857680,"kb_used_data":717108,"kb_used_omap":14,"kb_used_meta":2140465,"kb_avail":374629680,"statfs":{"total":386547056640,"available":383620792320,"internally_reserved":0,"allocated":734318592,"data_stored":733561784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2191836726},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,8,15,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18784256,"num_objects":-2293,"num_object_clones":0,"num_object_copies":-9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2293,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004743"},"pg_stats":[{"pgid":"1.f","version":"35'9690","reported_seq":9784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.256132+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:56.256132+0000","last_peered":"2024-09-11T05:03:56.256132+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:56.256132+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":2824,"num_object_clones":0,"num_object_copies":11296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9888","reported_seq":10002,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.454728+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:59.454728+0000","last_peered":"2024-09-11T05:03:59.454728+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:59.454728+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9888,"log_dups_size":0,"ondisk_log_size":9888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23344288,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9415","reported_seq":9519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.791473+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:56.791473+0000","last_peered":"2024-09-11T05:03:56.791473+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:56.791473+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9415,"log_dups_size":0,"ondisk_log_size":9415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22485608,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9414,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":535}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9691","reported_seq":9799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.476722+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:59.476722+0000","last_peered":"2024-09-11T05:03:59.476722+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:59.476722+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9691,"log_dups_size":0,"ondisk_log_size":9691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320288,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9744","reported_seq":9865,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.328276+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:56.328276+0000","last_peered":"2024-09-11T05:03:56.328276+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:56.328276+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9744,"log_dups_size":0,"ondisk_log_size":9744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22828192,"num_objects":2787,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9742,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9825","reported_seq":9930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.231687+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:56.231687+0000","last_peered":"2024-09-11T05:03:56.231687+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:56.231687+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9825,"log_dups_size":0,"ondisk_log_size":9825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23060480,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9917","reported_seq":10011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.436633+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:59.436633+0000","last_peered":"2024-09-11T05:03:59.436633+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:59.436633+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041184,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9915,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9971","reported_seq":14827,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:56.676619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:56.676619+0000","last_peered":"2024-09-11T05:03:56.676619+0000","last_clean":"2024-09-11T05:03:56.676619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:56.676619+0000","last_undegraded":"2024-09-11T05:03:56.676619+0000","last_fullsized":"2024-09-11T05:03:56.676619+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9971,"log_dups_size":0,"ondisk_log_size":9971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23363584,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":681,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9969","reported_seq":10091,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.265381+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:56.265381+0000","last_peered":"2024-09-11T05:03:56.265381+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:56.265381+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23087424,"num_objects":2819,"num_object_clones":0,"num_object_copies":11276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9754","reported_seq":9866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.791366+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:56.791366+0000","last_peered":"2024-09-11T05:03:56.791366+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:56.791366+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22721696,"num_objects":2774,"num_object_clones":0,"num_object_copies":11096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":468}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9800","reported_seq":9916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.124101+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:56.124101+0000","last_peered":"2024-09-11T05:03:56.124101+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:56.124101+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9800,"log_dups_size":0,"ondisk_log_size":9800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23367408,"num_objects":2853,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9797,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9629","reported_seq":14229,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:56.470238+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:56.470238+0000","last_peered":"2024-09-11T05:03:56.470238+0000","last_clean":"2024-09-11T05:03:56.470238+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:56.470238+0000","last_undegraded":"2024-09-11T05:03:56.470238+0000","last_fullsized":"2024-09-11T05:03:56.470238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22568960,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3439,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9932","reported_seq":11187,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.889702+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:56.889702+0000","last_peered":"2024-09-11T05:03:56.889702+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:56.889702+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22862416,"num_objects":2791,"num_object_clones":0,"num_object_copies":11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9931,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9796","reported_seq":9901,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.326636+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:56.326636+0000","last_peered":"2024-09-11T05:03:56.326636+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:56.326636+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9796,"log_dups_size":0,"ondisk_log_size":9796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22901920,"num_objects":2796,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9794,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":532}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9724","reported_seq":9839,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.856811+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:56.856811+0000","last_peered":"2024-09-11T05:03:56.856811+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:56.856811+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9724,"log_dups_size":0,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21992608,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":382}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9955","reported_seq":10074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.540820+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:59.540820+0000","last_peered":"2024-09-11T05:03:59.540820+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:59.540820+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23114912,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":366195176,"num_objects":44707,"num_object_clones":0,"num_object_copies":178828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141196,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":732237824,"data_stored":732237824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156700,"ondisk_log_size":156700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725332,"kb_used_data":179960,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93646508,"statfs":{"total":96636764160,"available":95894024192,"internally_reserved":0,"allocated":184279040,"data_stored":184089838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":558427856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738452,"kb_used_data":179712,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93633388,"statfs":{"total":96636764160,"available":95880589312,"internally_reserved":0,"allocated":184025088,"data_stored":183835886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621396,"kb_used_data":180220,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93750444,"statfs":{"total":96636764160,"available":96000454656,"internally_reserved":0,"allocated":184545280,"data_stored":184356078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451735250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772500,"kb_used_data":177216,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93599340,"statfs":{"total":96636764160,"available":95845724160,"internally_reserved":0,"allocated":181469184,"data_stored":181279982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,5,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181149696,"data_stored":181149696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183767040,"data_stored":183767040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":183451648,"data_stored":183451648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183869440,"data_stored":183869440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:00.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:01.482 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:01.482 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:01.511 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-11T05:03:59.821745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":366195176,"num_objects":44707,"num_object_clones":0,"num_object_copies":178828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141196,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156700,"ondisk_log_size":156700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857680,"kb_used_data":717108,"kb_used_omap":14,"kb_used_meta":2140465,"kb_avail":374629680,"statfs":{"total":386547056640,"available":383620792320,"internally_reserved":0,"allocated":734318592,"data_stored":733561784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2191836726},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,8,15,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18784256,"num_objects":-2293,"num_object_clones":0,"num_object_copies":-9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2293,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004743"},"pg_stats":[{"pgid":"1.f","version":"35'9690","reported_seq":9784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.256132+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:03:56.256132+0000","last_peered":"2024-09-11T05:03:56.256132+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:03:56.256132+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":2824,"num_object_clones":0,"num_object_copies":11296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9888","reported_seq":10002,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.454728+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:59.454728+0000","last_peered":"2024-09-11T05:03:59.454728+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:59.454728+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9888,"log_dups_size":0,"ondisk_log_size":9888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23344288,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9415","reported_seq":9519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.791473+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:56.791473+0000","last_peered":"2024-09-11T05:03:56.791473+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:56.791473+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9415,"log_dups_size":0,"ondisk_log_size":9415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22485608,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9414,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":535}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9691","reported_seq":9799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.476722+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:59.476722+0000","last_peered":"2024-09-11T05:03:59.476722+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:59.476722+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9691,"log_dups_size":0,"ondisk_log_size":9691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320288,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9744","reported_seq":9865,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.328276+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:03:56.328276+0000","last_peered":"2024-09-11T05:03:56.328276+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:03:56.328276+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9744,"log_dups_size":0,"ondisk_log_size":9744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22828192,"num_objects":2787,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9742,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9825","reported_seq":9930,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.231687+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:03:56.231687+0000","last_peered":"2024-09-11T05:03:56.231687+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:03:56.231687+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9825,"log_dups_size":0,"ondisk_log_size":9825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23060480,"num_objects":2815,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9917","reported_seq":10011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.436633+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:59.436633+0000","last_peered":"2024-09-11T05:03:59.436633+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:59.436633+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041184,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9915,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9971","reported_seq":14827,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:56.676619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:56.676619+0000","last_peered":"2024-09-11T05:03:56.676619+0000","last_clean":"2024-09-11T05:03:56.676619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:56.676619+0000","last_undegraded":"2024-09-11T05:03:56.676619+0000","last_fullsized":"2024-09-11T05:03:56.676619+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9971,"log_dups_size":0,"ondisk_log_size":9971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23363584,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":681,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9969","reported_seq":10091,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.265381+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:03:56.265381+0000","last_peered":"2024-09-11T05:03:56.265381+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:03:56.265381+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23087424,"num_objects":2819,"num_object_clones":0,"num_object_copies":11276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9754","reported_seq":9866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.791366+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:56.791366+0000","last_peered":"2024-09-11T05:03:56.791366+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:56.791366+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22721696,"num_objects":2774,"num_object_clones":0,"num_object_copies":11096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":468}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9800","reported_seq":9916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.124101+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:03:56.124101+0000","last_peered":"2024-09-11T05:03:56.124101+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:03:56.124101+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9800,"log_dups_size":0,"ondisk_log_size":9800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23367408,"num_objects":2853,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9797,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9629","reported_seq":14229,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:56.470238+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:56.470238+0000","last_peered":"2024-09-11T05:03:56.470238+0000","last_clean":"2024-09-11T05:03:56.470238+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:56.470238+0000","last_undegraded":"2024-09-11T05:03:56.470238+0000","last_fullsized":"2024-09-11T05:03:56.470238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22568960,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3439,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9932","reported_seq":11187,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.889702+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:56.889702+0000","last_peered":"2024-09-11T05:03:56.889702+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:56.889702+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22862416,"num_objects":2791,"num_object_clones":0,"num_object_copies":11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9931,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9796","reported_seq":9901,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.326636+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:03:56.326636+0000","last_peered":"2024-09-11T05:03:56.326636+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:03:56.326636+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9796,"log_dups_size":0,"ondisk_log_size":9796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22901920,"num_objects":2796,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9794,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":532}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9724","reported_seq":9839,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.856811+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:56.856811+0000","last_peered":"2024-09-11T05:03:56.856811+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:56.856811+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9724,"log_dups_size":0,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21992608,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":382}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9955","reported_seq":10074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.540820+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:59.540820+0000","last_peered":"2024-09-11T05:03:59.540820+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:59.540820+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23114912,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":366195176,"num_objects":44707,"num_object_clones":0,"num_object_copies":178828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141196,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":732237824,"data_stored":732237824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156700,"ondisk_log_size":156700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725332,"kb_used_data":179960,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93646508,"statfs":{"total":96636764160,"available":95894024192,"internally_reserved":0,"allocated":184279040,"data_stored":184089838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":558427856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738452,"kb_used_data":179712,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93633388,"statfs":{"total":96636764160,"available":95880589312,"internally_reserved":0,"allocated":184025088,"data_stored":183835886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621396,"kb_used_data":180220,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93750444,"statfs":{"total":96636764160,"available":96000454656,"internally_reserved":0,"allocated":184545280,"data_stored":184356078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451735250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772500,"kb_used_data":177216,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93599340,"statfs":{"total":96636764160,"available":95845724160,"internally_reserved":0,"allocated":181469184,"data_stored":181279982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,5,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181149696,"data_stored":181149696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183767040,"data_stored":183767040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":183451648,"data_stored":183451648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183869440,"data_stored":183869440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:01.512 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9690", 'reported_seq': 9784, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.256132+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:03:56.256132+0000', 'last_peered': '2024-09-11T05:03:56.256132+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:03:56.256132+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9690, 'log_dups_size': 0, 'ondisk_log_size': 9690, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23134208, 'num_objects': 2824, 'num_object_clones': 0, 'num_object_copies': 11296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9690, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 533}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9888", 'reported_seq': 10002, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:59.454728+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:03:59.454728+0000', 'last_peered': '2024-09-11T05:03:59.454728+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:03:59.454728+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9888, 'log_dups_size': 0, 'ondisk_log_size': 9888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23344288, 'num_objects': 2850, 'num_object_clones': 0, 'num_object_copies': 11400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9886, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9415", 'reported_seq': 9519, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.791473+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:03:56.791473+0000', 'last_peered': '2024-09-11T05:03:56.791473+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:03:56.791473+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9415, 'log_dups_size': 0, 'ondisk_log_size': 9415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22485608, 'num_objects': 2746, 'num_object_clones': 0, 'num_object_copies': 10984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2746, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9414, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 535}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9691", 'reported_seq': 9799, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:59.476722+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:03:59.476722+0000', 'last_peered': '2024-09-11T05:03:59.476722+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:03:59.476722+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9691, 'log_dups_size': 0, 'ondisk_log_size': 9691, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22320288, 'num_objects': 2725, 'num_object_clones': 0, 'num_object_copies': 10900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9689, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:01.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9744", 'reported_seq': 9865, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.328276+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:03:56.328276+0000', 'last_peered': '2024-09-11T05:03:56.328276+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:03:56.328276+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9744, 'log_dups_size': 0, 'ondisk_log_size': 9744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22828192, 'num_objects': 2787, 'num_object_clones': 0, 'num_object_copies': 11148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9742, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 566}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9825", 'reported_seq': 9930, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.231687+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:03:56.231687+0000', 'last_peered': '2024-09-11T05:03:56.231687+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:03:56.231687+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9825, 'log_dups_size': 0, 'ondisk_log_size': 9825, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23060480, 'num_objects': 2815, 'num_object_clones': 0, 'num_object_copies': 11260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9825, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 521}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9917", 'reported_seq': 10011, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:59.436633+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:03:59.436633+0000', 'last_peered': '2024-09-11T05:03:59.436633+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:03:59.436633+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9917, 'log_dups_size': 0, 'ondisk_log_size': 9917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23041184, 'num_objects': 2813, 'num_object_clones': 0, 'num_object_copies': 11252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9915, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 506}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9969", 'reported_seq': 10091, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.265381+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:03:56.265381+0000', 'last_peered': '2024-09-11T05:03:56.265381+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:03:56.265381+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9969, 'log_dups_size': 0, 'ondisk_log_size': 9969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23087424, 'num_objects': 2819, 'num_object_clones': 0, 'num_object_copies': 11276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9966, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 453}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9754", 'reported_seq': 9866, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.791366+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:03:56.791366+0000', 'last_peered': '2024-09-11T05:03:56.791366+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:03:56.791366+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9754, 'log_dups_size': 0, 'ondisk_log_size': 9754, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22721696, 'num_objects': 2774, 'num_object_clones': 0, 'num_object_copies': 11096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9752, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 468}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:01.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9800", 'reported_seq': 9916, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.124101+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:03:56.124101+0000', 'last_peered': '2024-09-11T05:03:56.124101+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:03:56.124101+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9800, 'log_dups_size': 0, 'ondisk_log_size': 9800, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23367408, 'num_objects': 2853, 'num_object_clones': 0, 'num_object_copies': 11412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9797, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 555}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:01.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:01.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9932", 'reported_seq': 11187, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.889702+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:03:56.889702+0000', 'last_peered': '2024-09-11T05:03:56.889702+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:03:56.889702+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9932, 'log_dups_size': 0, 'ondisk_log_size': 9932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22862416, 'num_objects': 2791, 'num_object_clones': 0, 'num_object_copies': 11164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1748, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9931, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 586, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:01.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:01.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9796", 'reported_seq': 9901, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.326636+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:03:56.326636+0000', 'last_peered': '2024-09-11T05:03:56.326636+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:03:56.326636+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9796, 'log_dups_size': 0, 'ondisk_log_size': 9796, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22901920, 'num_objects': 2796, 'num_object_clones': 0, 'num_object_copies': 11184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9794, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:01.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:01.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9724", 'reported_seq': 9839, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:56.856811+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:03:56.856811+0000', 'last_peered': '2024-09-11T05:03:56.856811+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:03:56.856811+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9724, 'log_dups_size': 0, 'ondisk_log_size': 9724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21992608, 'num_objects': 2685, 'num_object_clones': 0, 'num_object_copies': 10740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9722, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 382}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:01.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:01.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9955", 'reported_seq': 10074, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:03:59.540820+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:03:59.540820+0000', 'last_peered': '2024-09-11T05:03:59.540820+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:03:59.540820+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9955, 'log_dups_size': 0, 'ondisk_log_size': 9955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23114912, 'num_objects': 2822, 'num_object_clones': 0, 'num_object_copies': 11288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9953, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:01.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:02.376 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31439/177388 objects degraded (17.723%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":569,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44347,"data_bytes":363246056,"bytes_used":2964885504,"bytes_avail":383582171136,"bytes_total":386547056640,"degraded_objects":31439,"degraded_total":177388,"degraded_ratio":0.17723295826098723,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:02.399 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:02.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:03.005 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:03.030 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-11T05:04:01.822566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":363246056,"num_objects":44347,"num_object_clones":0,"num_object_copies":177388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141556,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157060,"ondisk_log_size":157060,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2895396,"kb_used_data":713156,"kb_used_omap":14,"kb_used_meta":2182129,"kb_avail":374591964,"statfs":{"total":386547056640,"available":383582171136,"internally_reserved":0,"allocated":730271744,"data_stored":729514936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2234500662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1,9,13,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19546112,"num_objects":-2386,"num_object_clones":0,"num_object_copies":-9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004858"},"pg_stats":[{"pgid":"1.f","version":"35'9746","reported_seq":9840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.374531+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:01.374531+0000","last_peered":"2024-09-11T05:04:01.374531+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:01.374531+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9746,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9888","reported_seq":10002,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.454728+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:03:59.454728+0000","last_peered":"2024-09-11T05:03:59.454728+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:03:59.454728+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9888,"log_dups_size":0,"ondisk_log_size":9888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23344288,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9415","reported_seq":9519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.791473+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:03:56.791473+0000","last_peered":"2024-09-11T05:03:56.791473+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:03:56.791473+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9415,"log_dups_size":0,"ondisk_log_size":9415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22485608,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9414,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":535}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9691","reported_seq":9799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.476722+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:03:59.476722+0000","last_peered":"2024-09-11T05:03:59.476722+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:03:59.476722+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9691,"log_dups_size":0,"ondisk_log_size":9691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320288,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9806","reported_seq":9929,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.259171+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:01.259171+0000","last_peered":"2024-09-11T05:04:01.259171+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:01.259171+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320288,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":504}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9878","reported_seq":9983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.176855+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:01.176855+0000","last_peered":"2024-09-11T05:04:01.176855+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:01.176855+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22626304,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9878,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":468}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9917","reported_seq":10011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.436633+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:03:59.436633+0000","last_peered":"2024-09-11T05:03:59.436633+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:03:59.436633+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041184,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9915,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9971","reported_seq":14827,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:56.676619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:03:56.676619+0000","last_peered":"2024-09-11T05:03:56.676619+0000","last_clean":"2024-09-11T05:03:56.676619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:03:56.676619+0000","last_undegraded":"2024-09-11T05:03:56.676619+0000","last_fullsized":"2024-09-11T05:03:56.676619+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":9971,"log_dups_size":0,"ondisk_log_size":9971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23363584,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":681,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10030","reported_seq":10153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.320017+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:01.320017+0000","last_peered":"2024-09-11T05:04:01.320017+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:01.320017+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22587712,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9754","reported_seq":9866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.791366+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:03:56.791366+0000","last_peered":"2024-09-11T05:03:56.791366+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:03:56.791366+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22721696,"num_objects":2774,"num_object_clones":0,"num_object_copies":11096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":468}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9875","reported_seq":9992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.340020+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:01.340020+0000","last_peered":"2024-09-11T05:04:01.340020+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:01.340020+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9875,"log_dups_size":0,"ondisk_log_size":9875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22753008,"num_objects":2778,"num_object_clones":0,"num_object_copies":11112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9872,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9629","reported_seq":14229,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:03:56.470238+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:03:56.470238+0000","last_peered":"2024-09-11T05:03:56.470238+0000","last_clean":"2024-09-11T05:03:56.470238+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:03:56.470238+0000","last_undegraded":"2024-09-11T05:03:56.470238+0000","last_fullsized":"2024-09-11T05:03:56.470238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22568960,"num_objects":2755,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3439,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9932","reported_seq":11187,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.889702+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:03:56.889702+0000","last_peered":"2024-09-11T05:03:56.889702+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:03:56.889702+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22862416,"num_objects":2791,"num_object_clones":0,"num_object_copies":11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9931,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9849","reported_seq":9955,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.237559+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:01.237559+0000","last_peered":"2024-09-11T05:04:01.237559+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:01.237559+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467744,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9847,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":479}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9724","reported_seq":9839,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:56.856811+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:03:56.856811+0000","last_peered":"2024-09-11T05:03:56.856811+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:03:56.856811+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9724,"log_dups_size":0,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21992608,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":382}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9955","reported_seq":10074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:03:59.540820+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:03:59.540820+0000","last_peered":"2024-09-11T05:03:59.540820+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:03:59.540820+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23114912,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":363246056,"num_objects":44347,"num_object_clones":0,"num_object_copies":177388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141556,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":728068096,"data_stored":728068096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157060,"ondisk_log_size":157060,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763048,"kb_used_data":176008,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93608792,"statfs":{"total":96636764160,"available":95855403008,"internally_reserved":0,"allocated":180232192,"data_stored":180042990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":601091792},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738452,"kb_used_data":179712,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93633388,"statfs":{"total":96636764160,"available":95880589312,"internally_reserved":0,"allocated":184025088,"data_stored":183835886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621396,"kb_used_data":180220,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93750444,"statfs":{"total":96636764160,"available":96000454656,"internally_reserved":0,"allocated":184545280,"data_stored":184356078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451735250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772500,"kb_used_data":177216,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93599340,"statfs":{"total":96636764160,"available":95845724160,"internally_reserved":0,"allocated":181469184,"data_stored":181279982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,5,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181149696,"data_stored":181149696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183767040,"data_stored":183767040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":183451648,"data_stored":183451648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179699712,"data_stored":179699712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:06.032 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:06.695 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:06.712 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-11T05:04:05.824317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":358003176,"num_objects":43707,"num_object_clones":0,"num_object_copies":174828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142196,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157700,"ondisk_log_size":157700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2915340,"kb_used_data":701160,"kb_used_omap":14,"kb_used_meta":2214065,"kb_avail":374572020,"statfs":{"total":386547056640,"available":383561748480,"internally_reserved":0,"allocated":717987840,"data_stored":717231032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2267203126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,16,10,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18571264,"num_objects":-2267,"num_object_clones":0,"num_object_copies":-9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2267,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005084"},"pg_stats":[{"pgid":"1.f","version":"35'9746","reported_seq":9840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.374531+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:01.374531+0000","last_peered":"2024-09-11T05:04:01.374531+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:01.374531+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9746,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9955","reported_seq":10069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.530038+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:04.530038+0000","last_peered":"2024-09-11T05:04:04.530038+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:04.530038+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22795424,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9478","reported_seq":9582,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.716045+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:01.716045+0000","last_peered":"2024-09-11T05:04:01.716045+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:01.716045+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9478,"log_dups_size":0,"ondisk_log_size":9478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21969512,"num_objects":2683,"num_object_clones":0,"num_object_copies":10732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9477,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9746","reported_seq":9854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.455020+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:04.455020+0000","last_peered":"2024-09-11T05:04:04.455020+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:04.455020+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21869728,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9806","reported_seq":9929,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.259171+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:01.259171+0000","last_peered":"2024-09-11T05:04:01.259171+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:01.259171+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320288,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":504}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9878","reported_seq":9983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.176855+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:01.176855+0000","last_peered":"2024-09-11T05:04:01.176855+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:01.176855+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22626304,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9878,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":468}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9978","reported_seq":10071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.382986+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:04.382986+0000","last_peered":"2024-09-11T05:04:04.382986+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:04.382986+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22541472,"num_objects":2752,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10043","reported_seq":14899,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:01.739287+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:01.739287+0000","last_peered":"2024-09-11T05:04:01.739287+0000","last_clean":"2024-09-11T05:04:01.739287+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:01.739287+0000","last_undegraded":"2024-09-11T05:04:01.739287+0000","last_fullsized":"2024-09-11T05:04:01.739287+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22773760,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":753,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10030","reported_seq":10153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.320017+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:01.320017+0000","last_peered":"2024-09-11T05:04:01.320017+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:01.320017+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22587712,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9821","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.961573+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:01.961573+0000","last_peered":"2024-09-11T05:04:01.961573+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:01.961573+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22172832,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":401}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9875","reported_seq":9992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.340020+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:01.340020+0000","last_peered":"2024-09-11T05:04:01.340020+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:01.340020+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9875,"log_dups_size":0,"ondisk_log_size":9875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22753008,"num_objects":2778,"num_object_clones":0,"num_object_copies":11112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9872,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9710","reported_seq":14310,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:01.805149+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:01.805149+0000","last_peered":"2024-09-11T05:04:01.805149+0000","last_clean":"2024-09-11T05:04:01.805149+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:01.805149+0000","last_undegraded":"2024-09-11T05:04:01.805149+0000","last_fullsized":"2024-09-11T05:04:01.805149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21905408,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3520,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9985","reported_seq":11343,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.970659+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:01.970659+0000","last_peered":"2024-09-11T05:04:01.970659+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:01.970659+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9985,"log_dups_size":0,"ondisk_log_size":9985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22428240,"num_objects":2738,"num_object_clones":0,"num_object_copies":10952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9984,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1041}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9849","reported_seq":9955,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.237559+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:01.237559+0000","last_peered":"2024-09-11T05:04:01.237559+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:01.237559+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467744,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9847,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":479}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9783","reported_seq":9901,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.973443+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:01.973443+0000","last_peered":"2024-09-11T05:04:01.973443+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:01.973443+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21509280,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":323}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10017","reported_seq":10136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.452792+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:04.452792+0000","last_peered":"2024-09-11T05:04:04.452792+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:04.452792+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22607008,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":358003176,"num_objects":43707,"num_object_clones":0,"num_object_copies":174828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142196,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715640832,"data_stored":715640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157700,"ondisk_log_size":157700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763048,"kb_used_data":176008,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93608792,"statfs":{"total":96636764160,"available":95855403008,"internally_reserved":0,"allocated":180232192,"data_stored":180042990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":601091792},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774700,"kb_used_data":175704,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93597140,"statfs":{"total":96636764160,"available":95843471360,"internally_reserved":0,"allocated":179920896,"data_stored":179731694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604848,"kb_used_data":176160,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93766992,"statfs":{"total":96636764160,"available":96017399808,"internally_reserved":0,"allocated":180387840,"data_stored":180198638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":438955730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772744,"kb_used_data":173288,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93599096,"statfs":{"total":96636764160,"available":95845474304,"internally_reserved":0,"allocated":177446912,"data_stored":177257710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177135616,"data_stored":177135616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179499008,"data_stored":179499008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179306496,"data_stored":179306496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179699712,"data_stored":179699712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:06.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:07.397 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:07.398 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:07.419 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-11T05:04:05.824317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":358003176,"num_objects":43707,"num_object_clones":0,"num_object_copies":174828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142196,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157700,"ondisk_log_size":157700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2915340,"kb_used_data":701160,"kb_used_omap":14,"kb_used_meta":2214065,"kb_avail":374572020,"statfs":{"total":386547056640,"available":383561748480,"internally_reserved":0,"allocated":717987840,"data_stored":717231032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2267203126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,16,10,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18571264,"num_objects":-2267,"num_object_clones":0,"num_object_copies":-9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2267,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005084"},"pg_stats":[{"pgid":"1.f","version":"35'9746","reported_seq":9840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.374531+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:01.374531+0000","last_peered":"2024-09-11T05:04:01.374531+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:01.374531+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9746,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9955","reported_seq":10069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.530038+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:04.530038+0000","last_peered":"2024-09-11T05:04:04.530038+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:04.530038+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22795424,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9478","reported_seq":9582,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.716045+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:01.716045+0000","last_peered":"2024-09-11T05:04:01.716045+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:01.716045+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9478,"log_dups_size":0,"ondisk_log_size":9478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21969512,"num_objects":2683,"num_object_clones":0,"num_object_copies":10732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9477,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9746","reported_seq":9854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.455020+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:04.455020+0000","last_peered":"2024-09-11T05:04:04.455020+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:04.455020+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21869728,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9806","reported_seq":9929,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.259171+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:01.259171+0000","last_peered":"2024-09-11T05:04:01.259171+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:01.259171+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320288,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":504}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9878","reported_seq":9983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.176855+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:01.176855+0000","last_peered":"2024-09-11T05:04:01.176855+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:01.176855+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22626304,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9878,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":468}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9978","reported_seq":10071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.382986+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:04.382986+0000","last_peered":"2024-09-11T05:04:04.382986+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:04.382986+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22541472,"num_objects":2752,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10043","reported_seq":14899,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:01.739287+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:01.739287+0000","last_peered":"2024-09-11T05:04:01.739287+0000","last_clean":"2024-09-11T05:04:01.739287+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:01.739287+0000","last_undegraded":"2024-09-11T05:04:01.739287+0000","last_fullsized":"2024-09-11T05:04:01.739287+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22773760,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":753,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10030","reported_seq":10153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.320017+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:01.320017+0000","last_peered":"2024-09-11T05:04:01.320017+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:01.320017+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22587712,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9821","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.961573+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:01.961573+0000","last_peered":"2024-09-11T05:04:01.961573+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:01.961573+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22172832,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":401}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9875","reported_seq":9992,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.340020+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:01.340020+0000","last_peered":"2024-09-11T05:04:01.340020+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:01.340020+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9875,"log_dups_size":0,"ondisk_log_size":9875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22753008,"num_objects":2778,"num_object_clones":0,"num_object_copies":11112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9872,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9710","reported_seq":14310,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:01.805149+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:01.805149+0000","last_peered":"2024-09-11T05:04:01.805149+0000","last_clean":"2024-09-11T05:04:01.805149+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:01.805149+0000","last_undegraded":"2024-09-11T05:04:01.805149+0000","last_fullsized":"2024-09-11T05:04:01.805149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21905408,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3520,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9985","reported_seq":11343,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.970659+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:01.970659+0000","last_peered":"2024-09-11T05:04:01.970659+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:01.970659+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9985,"log_dups_size":0,"ondisk_log_size":9985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22428240,"num_objects":2738,"num_object_clones":0,"num_object_copies":10952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9984,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1041}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9849","reported_seq":9955,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.237559+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:01.237559+0000","last_peered":"2024-09-11T05:04:01.237559+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:01.237559+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467744,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9847,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":479}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9783","reported_seq":9901,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:01.973443+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:01.973443+0000","last_peered":"2024-09-11T05:04:01.973443+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:01.973443+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21509280,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":323}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10017","reported_seq":10136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.452792+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:04.452792+0000","last_peered":"2024-09-11T05:04:04.452792+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:04.452792+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22607008,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":358003176,"num_objects":43707,"num_object_clones":0,"num_object_copies":174828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142196,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715640832,"data_stored":715640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157700,"ondisk_log_size":157700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763048,"kb_used_data":176008,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93608792,"statfs":{"total":96636764160,"available":95855403008,"internally_reserved":0,"allocated":180232192,"data_stored":180042990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":601091792},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774700,"kb_used_data":175704,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93597140,"statfs":{"total":96636764160,"available":95843471360,"internally_reserved":0,"allocated":179920896,"data_stored":179731694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604848,"kb_used_data":176160,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93766992,"statfs":{"total":96636764160,"available":96017399808,"internally_reserved":0,"allocated":180387840,"data_stored":180198638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":438955730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772744,"kb_used_data":173288,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93599096,"statfs":{"total":96636764160,"available":95845474304,"internally_reserved":0,"allocated":177446912,"data_stored":177257710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177135616,"data_stored":177135616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179499008,"data_stored":179499008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179306496,"data_stored":179306496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179699712,"data_stored":179699712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:07.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:07.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9746", 'reported_seq': 9840, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.374531+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:01.374531+0000', 'last_peered': '2024-09-11T05:04:01.374531+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:01.374531+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9746, 'log_dups_size': 0, 'ondisk_log_size': 9746, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22675456, 'num_objects': 2768, 'num_object_clones': 0, 'num_object_copies': 11072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2768, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9746, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 477}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9955", 'reported_seq': 10069, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:04.530038+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:04.530038+0000', 'last_peered': '2024-09-11T05:04:04.530038+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:04.530038+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9955, 'log_dups_size': 0, 'ondisk_log_size': 9955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22795424, 'num_objects': 2783, 'num_object_clones': 0, 'num_object_copies': 11132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9953, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9478", 'reported_seq': 9582, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.716045+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:01.716045+0000', 'last_peered': '2024-09-11T05:04:01.716045+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:01.716045+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9478, 'log_dups_size': 0, 'ondisk_log_size': 9478, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21969512, 'num_objects': 2683, 'num_object_clones': 0, 'num_object_copies': 10732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2683, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9477, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 472}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9746", 'reported_seq': 9854, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:04.455020+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:04.455020+0000', 'last_peered': '2024-09-11T05:04:04.455020+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:04.455020+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9746, 'log_dups_size': 0, 'ondisk_log_size': 9746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21869728, 'num_objects': 2670, 'num_object_clones': 0, 'num_object_copies': 10680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9744, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 409}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9806", 'reported_seq': 9929, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.259171+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:01.259171+0000', 'last_peered': '2024-09-11T05:04:01.259171+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:01.259171+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9806, 'log_dups_size': 0, 'ondisk_log_size': 9806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22320288, 'num_objects': 2725, 'num_object_clones': 0, 'num_object_copies': 10900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9804, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 504}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:07.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9878", 'reported_seq': 9983, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.176855+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:01.176855+0000', 'last_peered': '2024-09-11T05:04:01.176855+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:01.176855+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9878, 'log_dups_size': 0, 'ondisk_log_size': 9878, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22626304, 'num_objects': 2762, 'num_object_clones': 0, 'num_object_copies': 11048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9878, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 468}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9978", 'reported_seq': 10071, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:04.382986+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:04.382986+0000', 'last_peered': '2024-09-11T05:04:04.382986+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:04.382986+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9978, 'log_dups_size': 0, 'ondisk_log_size': 9978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22541472, 'num_objects': 2752, 'num_object_clones': 0, 'num_object_copies': 11008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9976, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 445}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10030", 'reported_seq': 10153, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.320017+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:01.320017+0000', 'last_peered': '2024-09-11T05:04:01.320017+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:01.320017+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22587712, 'num_objects': 2758, 'num_object_clones': 0, 'num_object_copies': 11032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10027, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9821", 'reported_seq': 9933, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.961573+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:01.961573+0000', 'last_peered': '2024-09-11T05:04:01.961573+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:01.961573+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9821, 'log_dups_size': 0, 'ondisk_log_size': 9821, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22172832, 'num_objects': 2707, 'num_object_clones': 0, 'num_object_copies': 10828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9819, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 401}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9875", 'reported_seq': 9992, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.340020+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:01.340020+0000', 'last_peered': '2024-09-11T05:04:01.340020+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:01.340020+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9875, 'log_dups_size': 0, 'ondisk_log_size': 9875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22753008, 'num_objects': 2778, 'num_object_clones': 0, 'num_object_copies': 11112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9872, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 480}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:07.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9985", 'reported_seq': 11343, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.970659+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:01.970659+0000', 'last_peered': '2024-09-11T05:04:01.970659+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:01.970659+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9985, 'log_dups_size': 0, 'ondisk_log_size': 9985, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22428240, 'num_objects': 2738, 'num_object_clones': 0, 'num_object_copies': 10952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1697, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9984, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 637, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1041}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:07.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:07.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9849", 'reported_seq': 9955, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.237559+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:01.237559+0000', 'last_peered': '2024-09-11T05:04:01.237559+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:01.237559+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9849, 'log_dups_size': 0, 'ondisk_log_size': 9849, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22467744, 'num_objects': 2743, 'num_object_clones': 0, 'num_object_copies': 10972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9847, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 479}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:07.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:07.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9783", 'reported_seq': 9901, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:01.973443+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:01.973443+0000', 'last_peered': '2024-09-11T05:04:01.973443+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:01.973443+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9783, 'log_dups_size': 0, 'ondisk_log_size': 9783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21509280, 'num_objects': 2626, 'num_object_clones': 0, 'num_object_copies': 10504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9781, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 323}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:07.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:07.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10017", 'reported_seq': 10136, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:04.452792+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:04.452792+0000', 'last_peered': '2024-09-11T05:04:04.452792+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:04.452792+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22607008, 'num_objects': 2760, 'num_object_clones': 0, 'num_object_copies': 11040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10015, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:07.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:08.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:08.203 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31332/171692 objects degraded (18.249%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":575,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42923,"data_bytes":351580648,"bytes_used":2968801280,"bytes_avail":383578255360,"bytes_total":386547056640,"degraded_objects":31332,"degraded_total":171692,"degraded_ratio":0.1824895743540759,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":233},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:08.224 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:08.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:08.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:08.953 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-11T05:04:07.825524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":351580648,"num_objects":42923,"num_object_clones":0,"num_object_copies":171692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42923,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142980,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158384,"ondisk_log_size":158384,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2899220,"kb_used_data":689220,"kb_used_omap":14,"kb_used_meta":2209905,"kb_avail":374588140,"statfs":{"total":386547056640,"available":383578255360,"internally_reserved":0,"allocated":705761280,"data_stored":705004472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2262943286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5,7,8,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22970368,"num_objects":-2804,"num_object_clones":0,"num_object_copies":-11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2804,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005477"},"pg_stats":[{"pgid":"1.f","version":"35'9792","reported_seq":9886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.295599+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:06.295599+0000","last_peered":"2024-09-11T05:04:06.295599+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:06.295599+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22298624,"num_objects":2722,"num_object_clones":0,"num_object_copies":10888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9955","reported_seq":10069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.530038+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:04.530038+0000","last_peered":"2024-09-11T05:04:04.530038+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:04.530038+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22795424,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9541","reported_seq":9646,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.967453+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:06.967453+0000","last_peered":"2024-09-11T05:04:06.967453+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:06.967453+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9541,"log_dups_size":0,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21453416,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9540,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9746","reported_seq":9854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.455020+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:04.455020+0000","last_peered":"2024-09-11T05:04:04.455020+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:04.455020+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21869728,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9872","reported_seq":9995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.355491+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:06.355491+0000","last_peered":"2024-09-11T05:04:06.355491+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:06.355491+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9872,"log_dups_size":0,"ondisk_log_size":9872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21779616,"num_objects":2659,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":438}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9948","reported_seq":10053,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.467963+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:06.467963+0000","last_peered":"2024-09-11T05:04:06.467963+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:06.467963+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22052864,"num_objects":2692,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9948,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9978","reported_seq":10071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.382986+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:04.382986+0000","last_peered":"2024-09-11T05:04:04.382986+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:04.382986+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22541472,"num_objects":2752,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10117","reported_seq":14974,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:06.839719+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:06.839719+0000","last_peered":"2024-09-11T05:04:06.839719+0000","last_clean":"2024-09-11T05:04:06.839719+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:06.839719+0000","last_undegraded":"2024-09-11T05:04:06.839719+0000","last_fullsized":"2024-09-11T05:04:06.839719+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22167552,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":827,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10089","reported_seq":10212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.359797+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:06.359797+0000","last_peered":"2024-09-11T05:04:06.359797+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:06.359797+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22104384,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9894","reported_seq":10006,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:07.147234+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:07.147234+0000","last_peered":"2024-09-11T05:04:07.147234+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:07.147234+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574816,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9892,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":328}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9944","reported_seq":10061,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.489226+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:06.489226+0000","last_peered":"2024-09-11T05:04:06.489226+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:06.489226+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22187760,"num_objects":2709,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9781","reported_seq":14382,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:06.752904+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:06.752904+0000","last_peered":"2024-09-11T05:04:06.752904+0000","last_clean":"2024-09-11T05:04:06.752904+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:06.752904+0000","last_undegraded":"2024-09-11T05:04:06.752904+0000","last_fullsized":"2024-09-11T05:04:06.752904+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9781,"log_dups_size":0,"ondisk_log_size":9781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21323776,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3591,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10040","reported_seq":11510,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:07.083886+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:07.083886+0000","last_peered":"2024-09-11T05:04:07.083886+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:07.083886+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21977680,"num_objects":2683,"num_object_clones":0,"num_object_copies":10732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9918","reported_seq":10023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.371527+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:06.371527+0000","last_peered":"2024-09-11T05:04:06.371527+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:06.371527+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21902496,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9916,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9852","reported_seq":9971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.924131+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:06.924131+0000","last_peered":"2024-09-11T05:04:06.924131+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:06.924131+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9852,"log_dups_size":0,"ondisk_log_size":9852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":254}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10017","reported_seq":10136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:04.452792+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:04.452792+0000","last_peered":"2024-09-11T05:04:04.452792+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:04.452792+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22607008,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":351580648,"num_objects":42923,"num_object_clones":0,"num_object_copies":171692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42923,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142980,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":703184896,"data_stored":703184896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158384,"ondisk_log_size":158384,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763220,"kb_used_data":172036,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93608620,"statfs":{"total":96636764160,"available":95855226880,"internally_reserved":0,"allocated":176164864,"data_stored":175975662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758204,"kb_used_data":171692,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93613636,"statfs":{"total":96636764160,"available":95860363264,"internally_reserved":0,"allocated":175812608,"data_stored":175623406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605052,"kb_used_data":172204,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93766788,"statfs":{"total":96636764160,"available":96017190912,"internally_reserved":0,"allocated":176336896,"data_stored":176147694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443215570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":772744,"kb_used_data":173288,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93599096,"statfs":{"total":96636764160,"available":95845474304,"internally_reserved":0,"allocated":177446912,"data_stored":177257710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177135616,"data_stored":177135616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":175357952,"data_stored":175357952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":175104000,"data_stored":175104000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":175587328,"data_stored":175587328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:11.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:12.640 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:12.665 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-11T05:04:11.827256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":346452456,"num_objects":42297,"num_object_clones":0,"num_object_copies":169188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143606,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158910,"ondisk_log_size":158910,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2883060,"kb_used_data":681396,"kb_used_omap":14,"kb_used_meta":2201585,"kb_avail":374604300,"statfs":{"total":386547056640,"available":383594803200,"internally_reserved":0,"allocated":697749504,"data_stored":696992696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2254423606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,10,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19742720,"num_objects":-2410,"num_object_clones":0,"num_object_copies":-9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005511"},"pg_stats":[{"pgid":"1.f","version":"35'9852","reported_seq":9945,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.444925+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:11.444925+0000","last_peered":"2024-09-11T05:04:11.444925+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:11.444925+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9852,"log_dups_size":0,"ondisk_log_size":9852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21807104,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9852,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":371}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10026","reported_seq":10138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.625681+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:09.625681+0000","last_peered":"2024-09-11T05:04:09.625681+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:09.625681+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22213792,"num_objects":2712,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10024,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9541","reported_seq":9646,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.967453+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:06.967453+0000","last_peered":"2024-09-11T05:04:06.967453+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:06.967453+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9541,"log_dups_size":0,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21453416,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9540,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9806","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.569523+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:09.569523+0000","last_peered":"2024-09-11T05:04:09.569523+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:09.569523+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21378208,"num_objects":2610,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9920","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.369727+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:11.369727+0000","last_peered":"2024-09-11T05:04:11.369727+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:11.369727+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9920,"log_dups_size":0,"ondisk_log_size":9920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21386400,"num_objects":2611,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9918,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10020","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.397725+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:11.397725+0000","last_peered":"2024-09-11T05:04:11.397725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:11.397725+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21463040,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10020,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10037","reported_seq":10131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.554143+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:09.554143+0000","last_peered":"2024-09-11T05:04:09.554143+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:09.554143+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22058144,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":386}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10117","reported_seq":14974,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:06.839719+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:06.839719+0000","last_peered":"2024-09-11T05:04:06.839719+0000","last_clean":"2024-09-11T05:04:06.839719+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:06.839719+0000","last_undegraded":"2024-09-11T05:04:06.839719+0000","last_fullsized":"2024-09-11T05:04:06.839719+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22167552,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":827,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10155","reported_seq":10278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.404090+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:11.404090+0000","last_peered":"2024-09-11T05:04:11.404090+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:11.404090+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21563712,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9894","reported_seq":10006,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:07.147234+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:07.147234+0000","last_peered":"2024-09-11T05:04:07.147234+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:07.147234+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574816,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9892,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":328}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10002","reported_seq":10119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.353774+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:11.353774+0000","last_peered":"2024-09-11T05:04:11.353774+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:11.353774+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21712624,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9781","reported_seq":14382,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:06.752904+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:06.752904+0000","last_peered":"2024-09-11T05:04:06.752904+0000","last_clean":"2024-09-11T05:04:06.752904+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:06.752904+0000","last_undegraded":"2024-09-11T05:04:06.752904+0000","last_fullsized":"2024-09-11T05:04:06.752904+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9781,"log_dups_size":0,"ondisk_log_size":9781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21323776,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3591,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10040","reported_seq":11510,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:07.083886+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:07.083886+0000","last_peered":"2024-09-11T05:04:07.083886+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:07.083886+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21977680,"num_objects":2683,"num_object_clones":0,"num_object_copies":10732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9989","reported_seq":10095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.390072+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:11.390072+0000","last_peered":"2024-09-11T05:04:11.390072+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:11.390072+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9989,"log_dups_size":0,"ondisk_log_size":9989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21320864,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9852","reported_seq":9971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.924131+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:06.924131+0000","last_peered":"2024-09-11T05:04:06.924131+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:06.924131+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9852,"log_dups_size":0,"ondisk_log_size":9852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":254}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10078","reported_seq":10197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.548885+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:09.548885+0000","last_peered":"2024-09-11T05:04:09.548885+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:09.548885+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22107296,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":411}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":346452456,"num_objects":42297,"num_object_clones":0,"num_object_copies":169188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143606,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":695119872,"data_stored":695119872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158910,"ondisk_log_size":158910,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763472,"kb_used_data":168128,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93608368,"statfs":{"total":96636764160,"available":95854968832,"internally_reserved":0,"allocated":172163072,"data_stored":171973870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758204,"kb_used_data":171692,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93613636,"statfs":{"total":96636764160,"available":95860363264,"internally_reserved":0,"allocated":175812608,"data_stored":175623406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605052,"kb_used_data":172204,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93766788,"statfs":{"total":96636764160,"available":96017190912,"internally_reserved":0,"allocated":176336896,"data_stored":176147694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443215570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756332,"kb_used_data":169372,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93615508,"statfs":{"total":96636764160,"available":95862280192,"internally_reserved":0,"allocated":173436928,"data_stored":173247726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173133824,"data_stored":173133824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":175357952,"data_stored":175357952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":175104000,"data_stored":175104000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171524096,"data_stored":171524096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:12.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:13.295 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:13.296 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:13.318 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-11T05:04:11.827256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":346452456,"num_objects":42297,"num_object_clones":0,"num_object_copies":169188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143606,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158910,"ondisk_log_size":158910,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2883060,"kb_used_data":681396,"kb_used_omap":14,"kb_used_meta":2201585,"kb_avail":374604300,"statfs":{"total":386547056640,"available":383594803200,"internally_reserved":0,"allocated":697749504,"data_stored":696992696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2254423606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,10,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19742720,"num_objects":-2410,"num_object_clones":0,"num_object_copies":-9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005511"},"pg_stats":[{"pgid":"1.f","version":"35'9852","reported_seq":9945,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.444925+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:11.444925+0000","last_peered":"2024-09-11T05:04:11.444925+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:11.444925+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9852,"log_dups_size":0,"ondisk_log_size":9852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21807104,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9852,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":371}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10026","reported_seq":10138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.625681+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:09.625681+0000","last_peered":"2024-09-11T05:04:09.625681+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:09.625681+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22213792,"num_objects":2712,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10024,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9541","reported_seq":9646,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.967453+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:06.967453+0000","last_peered":"2024-09-11T05:04:06.967453+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:06.967453+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9541,"log_dups_size":0,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21453416,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9540,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9806","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.569523+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:09.569523+0000","last_peered":"2024-09-11T05:04:09.569523+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:09.569523+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21378208,"num_objects":2610,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9920","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.369727+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:11.369727+0000","last_peered":"2024-09-11T05:04:11.369727+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:11.369727+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9920,"log_dups_size":0,"ondisk_log_size":9920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21386400,"num_objects":2611,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9918,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10020","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.397725+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:11.397725+0000","last_peered":"2024-09-11T05:04:11.397725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:11.397725+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21463040,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10020,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10037","reported_seq":10131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.554143+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:09.554143+0000","last_peered":"2024-09-11T05:04:09.554143+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:09.554143+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22058144,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":386}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10117","reported_seq":14974,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:06.839719+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:06.839719+0000","last_peered":"2024-09-11T05:04:06.839719+0000","last_clean":"2024-09-11T05:04:06.839719+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:06.839719+0000","last_undegraded":"2024-09-11T05:04:06.839719+0000","last_fullsized":"2024-09-11T05:04:06.839719+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22167552,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":827,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10155","reported_seq":10278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.404090+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:11.404090+0000","last_peered":"2024-09-11T05:04:11.404090+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:11.404090+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21563712,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9894","reported_seq":10006,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:07.147234+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:07.147234+0000","last_peered":"2024-09-11T05:04:07.147234+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:07.147234+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21574816,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9892,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":328}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10002","reported_seq":10119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.353774+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:11.353774+0000","last_peered":"2024-09-11T05:04:11.353774+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:11.353774+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21712624,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9781","reported_seq":14382,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:06.752904+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:06.752904+0000","last_peered":"2024-09-11T05:04:06.752904+0000","last_clean":"2024-09-11T05:04:06.752904+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:06.752904+0000","last_undegraded":"2024-09-11T05:04:06.752904+0000","last_fullsized":"2024-09-11T05:04:06.752904+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9781,"log_dups_size":0,"ondisk_log_size":9781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21323776,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3591,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10040","reported_seq":11510,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:07.083886+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:07.083886+0000","last_peered":"2024-09-11T05:04:07.083886+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:07.083886+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21977680,"num_objects":2683,"num_object_clones":0,"num_object_copies":10732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9989","reported_seq":10095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.390072+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:11.390072+0000","last_peered":"2024-09-11T05:04:11.390072+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:11.390072+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9989,"log_dups_size":0,"ondisk_log_size":9989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21320864,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9852","reported_seq":9971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:06.924131+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:06.924131+0000","last_peered":"2024-09-11T05:04:06.924131+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:06.924131+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9852,"log_dups_size":0,"ondisk_log_size":9852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20944032,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":254}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10078","reported_seq":10197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.548885+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:09.548885+0000","last_peered":"2024-09-11T05:04:09.548885+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:09.548885+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22107296,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":411}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":346452456,"num_objects":42297,"num_object_clones":0,"num_object_copies":169188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143606,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":695119872,"data_stored":695119872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158910,"ondisk_log_size":158910,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763472,"kb_used_data":168128,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93608368,"statfs":{"total":96636764160,"available":95854968832,"internally_reserved":0,"allocated":172163072,"data_stored":171973870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758204,"kb_used_data":171692,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93613636,"statfs":{"total":96636764160,"available":95860363264,"internally_reserved":0,"allocated":175812608,"data_stored":175623406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605052,"kb_used_data":172204,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93766788,"statfs":{"total":96636764160,"available":96017190912,"internally_reserved":0,"allocated":176336896,"data_stored":176147694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":443215570},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756332,"kb_used_data":169372,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93615508,"statfs":{"total":96636764160,"available":95862280192,"internally_reserved":0,"allocated":173436928,"data_stored":173247726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173133824,"data_stored":173133824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":175357952,"data_stored":175357952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":175104000,"data_stored":175104000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171524096,"data_stored":171524096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:13.319 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:13.319 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9852", 'reported_seq': 9945, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:11.444925+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:11.444925+0000', 'last_peered': '2024-09-11T05:04:11.444925+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:11.444925+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9852, 'log_dups_size': 0, 'ondisk_log_size': 9852, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21807104, 'num_objects': 2662, 'num_object_clones': 0, 'num_object_copies': 10648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9852, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 371}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:13.319 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:13.319 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10026", 'reported_seq': 10138, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:09.625681+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:09.625681+0000', 'last_peered': '2024-09-11T05:04:09.625681+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:09.625681+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22213792, 'num_objects': 2712, 'num_object_clones': 0, 'num_object_copies': 10848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10024, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 454}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:13.319 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:13.319 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9541", 'reported_seq': 9646, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:06.967453+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:06.967453+0000', 'last_peered': '2024-09-11T05:04:06.967453+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:06.967453+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9541, 'log_dups_size': 0, 'ondisk_log_size': 9541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21453416, 'num_objects': 2620, 'num_object_clones': 0, 'num_object_copies': 10480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2620, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9540, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 409}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9806", 'reported_seq': 9914, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:09.569523+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:09.569523+0000', 'last_peered': '2024-09-11T05:04:09.569523+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:09.569523+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9806, 'log_dups_size': 0, 'ondisk_log_size': 9806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21378208, 'num_objects': 2610, 'num_object_clones': 0, 'num_object_copies': 10440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9804, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 349}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9920", 'reported_seq': 10043, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:11.369727+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:11.369727+0000', 'last_peered': '2024-09-11T05:04:11.369727+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:11.369727+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9920, 'log_dups_size': 0, 'ondisk_log_size': 9920, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21386400, 'num_objects': 2611, 'num_object_clones': 0, 'num_object_copies': 10444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9918, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 390}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10020", 'reported_seq': 10125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:11.397725+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:11.397725+0000', 'last_peered': '2024-09-11T05:04:11.397725+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:11.397725+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21463040, 'num_objects': 2620, 'num_object_clones': 0, 'num_object_copies': 10480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10020, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 326}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10037", 'reported_seq': 10131, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:09.554143+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:09.554143+0000', 'last_peered': '2024-09-11T05:04:09.554143+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:09.554143+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22058144, 'num_objects': 2693, 'num_object_clones': 0, 'num_object_copies': 10772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10035, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 386}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:13.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10155", 'reported_seq': 10278, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:11.404090+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:11.404090+0000', 'last_peered': '2024-09-11T05:04:11.404090+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:11.404090+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21563712, 'num_objects': 2633, 'num_object_clones': 0, 'num_object_copies': 10532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10152, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 267}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9894", 'reported_seq': 10006, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:07.147234+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:07.147234+0000', 'last_peered': '2024-09-11T05:04:07.147234+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:07.147234+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9894, 'log_dups_size': 0, 'ondisk_log_size': 9894, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21574816, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9892, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 328}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10002", 'reported_seq': 10119, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:11.353774+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:11.353774+0000', 'last_peered': '2024-09-11T05:04:11.353774+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:11.353774+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21712624, 'num_objects': 2651, 'num_object_clones': 0, 'num_object_copies': 10604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9999, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 353}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10040", 'reported_seq': 11510, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:07.083886+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:07.083886+0000', 'last_peered': '2024-09-11T05:04:07.083886+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:07.083886+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21977680, 'num_objects': 2683, 'num_object_clones': 0, 'num_object_copies': 10732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1641, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10039, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 693, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9989", 'reported_seq': 10095, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:11.390072+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:11.390072+0000', 'last_peered': '2024-09-11T05:04:11.390072+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:11.390072+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9989, 'log_dups_size': 0, 'ondisk_log_size': 9989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21320864, 'num_objects': 2603, 'num_object_clones': 0, 'num_object_copies': 10412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9987, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:13.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9852", 'reported_seq': 9971, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:06.924131+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:06.924131+0000', 'last_peered': '2024-09-11T05:04:06.924131+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:06.924131+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9852, 'log_dups_size': 0, 'ondisk_log_size': 9852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20944032, 'num_objects': 2557, 'num_object_clones': 0, 'num_object_copies': 10228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9850, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:13.322 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:13.322 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10078", 'reported_seq': 10197, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:09.548885+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:09.548885+0000', 'last_peered': '2024-09-11T05:04:09.548885+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:09.548885+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22107296, 'num_objects': 2699, 'num_object_clones': 0, 'num_object_copies': 10796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10076, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 411}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:13.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:14.105 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:14.105 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31287/167848 objects degraded (18.640%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":581,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41962,"data_bytes":343708136,"bytes_used":2952736768,"bytes_avail":383594319872,"bytes_total":386547056640,"degraded_objects":31287,"degraded_total":167848,"degraded_ratio":0.18640079119203087,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:14.127 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:14.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:14.808 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:14.808 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:14.832 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-11T05:04:13.828123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":343708136,"num_objects":41962,"num_object_clones":0,"num_object_copies":167848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143941,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1021,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159145,"ondisk_log_size":159145,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2883532,"kb_used_data":673544,"kb_used_omap":14,"kb_used_meta":2209905,"kb_avail":374603828,"statfs":{"total":386547056640,"available":383594319872,"internally_reserved":0,"allocated":689709056,"data_stored":688952248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2262943286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,5,12,13,15,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19537920,"num_objects":-2385,"num_object_clones":0,"num_object_copies":-9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005557"},"pg_stats":[{"pgid":"1.f","version":"35'9852","reported_seq":9945,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.444925+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:11.444925+0000","last_peered":"2024-09-11T05:04:11.444925+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:11.444925+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9852,"log_dups_size":0,"ondisk_log_size":9852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21807104,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9852,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":371}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10026","reported_seq":10138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.625681+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:09.625681+0000","last_peered":"2024-09-11T05:04:09.625681+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:09.625681+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22213792,"num_objects":2712,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10024,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9600","reported_seq":9705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:12.029897+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:12.029897+0000","last_peered":"2024-09-11T05:04:12.029897+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:12.029897+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9600,"log_dups_size":0,"ondisk_log_size":9600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20970088,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9599,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":350}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9806","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.569523+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:09.569523+0000","last_peered":"2024-09-11T05:04:09.569523+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:09.569523+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21378208,"num_objects":2610,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9920","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.369727+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:11.369727+0000","last_peered":"2024-09-11T05:04:11.369727+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:11.369727+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9920,"log_dups_size":0,"ondisk_log_size":9920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21386400,"num_objects":2611,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9918,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10020","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.397725+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:11.397725+0000","last_peered":"2024-09-11T05:04:11.397725+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:11.397725+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21463040,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10020,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10037","reported_seq":10131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.554143+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:09.554143+0000","last_peered":"2024-09-11T05:04:09.554143+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:09.554143+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22058144,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":386}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10179","reported_seq":15036,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:12.063376+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:12.063376+0000","last_peered":"2024-09-11T05:04:12.063376+0000","last_clean":"2024-09-11T05:04:12.063376+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:12.063376+0000","last_undegraded":"2024-09-11T05:04:12.063376+0000","last_fullsized":"2024-09-11T05:04:12.063376+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21659648,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":889,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10155","reported_seq":10278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.404090+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:11.404090+0000","last_peered":"2024-09-11T05:04:11.404090+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:11.404090+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21563712,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9937","reported_seq":10048,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.966054+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:11.966054+0000","last_peered":"2024-09-11T05:04:11.966054+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:11.966054+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21222560,"num_objects":2591,"num_object_clones":0,"num_object_copies":10364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9935,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":285}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10002","reported_seq":10119,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.353774+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:11.353774+0000","last_peered":"2024-09-11T05:04:11.353774+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:11.353774+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21712624,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9822","reported_seq":14422,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:12.001332+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:12.001332+0000","last_peered":"2024-09-11T05:04:12.001332+0000","last_clean":"2024-09-11T05:04:12.001332+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:12.001332+0000","last_undegraded":"2024-09-11T05:04:12.001332+0000","last_fullsized":"2024-09-11T05:04:12.001332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9822,"log_dups_size":0,"ondisk_log_size":9822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20987904,"num_objects":2562,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3632,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10103","reported_seq":11663,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:12.210688+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:12.210688+0000","last_peered":"2024-09-11T05:04:12.210688+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:12.210688+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21461584,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":738,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9989","reported_seq":10095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:11.390072+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:11.390072+0000","last_peered":"2024-09-11T05:04:11.390072+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:11.390072+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9989,"log_dups_size":0,"ondisk_log_size":9989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21320864,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9919","reported_seq":10037,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:12.070851+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:12.070851+0000","last_peered":"2024-09-11T05:04:12.070851+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:12.070851+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9919,"log_dups_size":0,"ondisk_log_size":9919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20395168,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9917,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10078","reported_seq":10197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:09.548885+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:09.548885+0000","last_peered":"2024-09-11T05:04:09.548885+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:09.548885+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22107296,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":411}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":343708136,"num_objects":41962,"num_object_clones":0,"num_object_copies":167848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143941,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1021,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":686817280,"data_stored":686817280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159145,"ondisk_log_size":159145,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763472,"kb_used_data":168128,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93608368,"statfs":{"total":96636764160,"available":95854968832,"internally_reserved":0,"allocated":172163072,"data_stored":171973870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758468,"kb_used_data":167796,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93613372,"statfs":{"total":96636764160,"available":95860092928,"internally_reserved":0,"allocated":171823104,"data_stored":171633902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605260,"kb_used_data":168248,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93766580,"statfs":{"total":96636764160,"available":96016977920,"internally_reserved":0,"allocated":172285952,"data_stored":172096750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":447475410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,5,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756332,"kb_used_data":169372,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93615508,"statfs":{"total":96636764160,"available":95862280192,"internally_reserved":0,"allocated":173436928,"data_stored":173247726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173133824,"data_stored":173133824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171184128,"data_stored":171184128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170975232,"data_stored":170975232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171524096,"data_stored":171524096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:17.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:18.450 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:18.451 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:18.478 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-11T05:04:17.829951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":335581672,"num_objects":40970,"num_object_clones":0,"num_object_copies":163880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144933,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159737,"ondisk_log_size":159737,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2884712,"kb_used_data":658048,"kb_used_omap":14,"kb_used_meta":2226545,"kb_avail":374602648,"statfs":{"total":386547056640,"available":383593111552,"internally_reserved":0,"allocated":673841152,"data_stored":673084344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2279982646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,11,11,4,12,12,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22421504,"num_objects":-2737,"num_object_clones":0,"num_object_copies":-10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2737,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005633"},"pg_stats":[{"pgid":"1.f","version":"35'9907","reported_seq":10001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.388073+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:16.388073+0000","last_peered":"2024-09-11T05:04:16.388073+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:16.388073+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9907,"log_dups_size":0,"ondisk_log_size":9907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21356544,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10096","reported_seq":10209,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.632128+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:14.632128+0000","last_peered":"2024-09-11T05:04:14.632128+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:14.632128+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21640352,"num_objects":2642,"num_object_clones":0,"num_object_copies":10568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10094,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9661","reported_seq":9765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.159139+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:17.159139+0000","last_peered":"2024-09-11T05:04:17.159139+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:17.159139+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20470376,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9660,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":289}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9850","reported_seq":9958,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.684833+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:14.684833+0000","last_peered":"2024-09-11T05:04:14.684833+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:14.684833+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21017760,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":305}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9987","reported_seq":10110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.403456+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:16.403456+0000","last_peered":"2024-09-11T05:04:16.403456+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:16.403456+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9987,"log_dups_size":0,"ondisk_log_size":9987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20837536,"num_objects":2544,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9985,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10086","reported_seq":10191,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.289335+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:16.289335+0000","last_peered":"2024-09-11T05:04:16.289335+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:16.289335+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20922368,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10106","reported_seq":10200,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.527224+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:14.527224+0000","last_peered":"2024-09-11T05:04:14.527224+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:14.527224+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21492896,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10104,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":317}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10253","reported_seq":15110,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:17.106062+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:17.106062+0000","last_peered":"2024-09-11T05:04:17.106062+0000","last_clean":"2024-09-11T05:04:17.106062+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:17.106062+0000","last_undegraded":"2024-09-11T05:04:17.106062+0000","last_fullsized":"2024-09-11T05:04:17.106062+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":963,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10215","reported_seq":10338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.166810+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:16.166810+0000","last_peered":"2024-09-11T05:04:16.166810+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:16.166810+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21072192,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10009","reported_seq":10121,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.240571+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:17.240571+0000","last_peered":"2024-09-11T05:04:17.240571+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:17.240571+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10007,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":213}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10068","reported_seq":10185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.502513+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:16.502513+0000","last_peered":"2024-09-11T05:04:16.502513+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:16.502513+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21171952,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10065,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9880","reported_seq":14481,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:16.751387+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:16.751387+0000","last_peered":"2024-09-11T05:04:16.751387+0000","last_clean":"2024-09-11T05:04:16.751387+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:16.751387+0000","last_undegraded":"2024-09-11T05:04:16.751387+0000","last_fullsized":"2024-09-11T05:04:16.751387+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20512768,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10150","reported_seq":11801,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.271033+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:17.271033+0000","last_peered":"2024-09-11T05:04:17.271033+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:17.271033+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21076560,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10149,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1023}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10052","reported_seq":10158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.466317+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:16.466317+0000","last_peered":"2024-09-11T05:04:16.466317+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:16.466317+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20804768,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":276}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9973","reported_seq":10092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.257203+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:17.257203+0000","last_peered":"2024-09-11T05:04:17.257203+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:17.257203+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19952800,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":133}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10144","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.753741+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:14.753741+0000","last_peered":"2024-09-11T05:04:14.753741+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:14.753741+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21566624,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":335581672,"num_objects":40970,"num_object_clones":0,"num_object_copies":163880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144933,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":670855168,"data_stored":670855168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159737,"ondisk_log_size":159737,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763764,"kb_used_data":164232,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93608076,"statfs":{"total":96636764160,"available":95854669824,"internally_reserved":0,"allocated":168173568,"data_stored":167984366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758812,"kb_used_data":163972,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93613028,"statfs":{"total":96636764160,"available":95859740672,"internally_reserved":0,"allocated":167907328,"data_stored":167718126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,3,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605540,"kb_used_data":164372,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93766300,"statfs":{"total":96636764160,"available":96016691200,"internally_reserved":0,"allocated":168316928,"data_stored":168127726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451735250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,6,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756596,"kb_used_data":165472,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93615244,"statfs":{"total":96636764160,"available":95862009856,"internally_reserved":0,"allocated":169443328,"data_stored":169254126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169086976,"data_stored":169086976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167194624,"data_stored":167194624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166965248,"data_stored":166965248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167608320,"data_stored":167608320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:18.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:19.093 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:19.094 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:19.121 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-11T05:04:17.829951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":335581672,"num_objects":40970,"num_object_clones":0,"num_object_copies":163880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144933,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159737,"ondisk_log_size":159737,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2884712,"kb_used_data":658048,"kb_used_omap":14,"kb_used_meta":2226545,"kb_avail":374602648,"statfs":{"total":386547056640,"available":383593111552,"internally_reserved":0,"allocated":673841152,"data_stored":673084344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2279982646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,11,11,4,12,12,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22421504,"num_objects":-2737,"num_object_clones":0,"num_object_copies":-10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2737,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005633"},"pg_stats":[{"pgid":"1.f","version":"35'9907","reported_seq":10001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.388073+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:16.388073+0000","last_peered":"2024-09-11T05:04:16.388073+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:16.388073+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9907,"log_dups_size":0,"ondisk_log_size":9907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21356544,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10096","reported_seq":10209,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.632128+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:14.632128+0000","last_peered":"2024-09-11T05:04:14.632128+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:14.632128+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21640352,"num_objects":2642,"num_object_clones":0,"num_object_copies":10568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10094,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9661","reported_seq":9765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.159139+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:17.159139+0000","last_peered":"2024-09-11T05:04:17.159139+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:17.159139+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20470376,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9660,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":289}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9850","reported_seq":9958,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.684833+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:14.684833+0000","last_peered":"2024-09-11T05:04:14.684833+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:14.684833+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21017760,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":305}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9987","reported_seq":10110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.403456+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:16.403456+0000","last_peered":"2024-09-11T05:04:16.403456+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:16.403456+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9987,"log_dups_size":0,"ondisk_log_size":9987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20837536,"num_objects":2544,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9985,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10086","reported_seq":10191,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.289335+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:16.289335+0000","last_peered":"2024-09-11T05:04:16.289335+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:16.289335+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20922368,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10106","reported_seq":10200,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.527224+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:14.527224+0000","last_peered":"2024-09-11T05:04:14.527224+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:14.527224+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21492896,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10104,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":317}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10253","reported_seq":15110,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:17.106062+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:17.106062+0000","last_peered":"2024-09-11T05:04:17.106062+0000","last_clean":"2024-09-11T05:04:17.106062+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:17.106062+0000","last_undegraded":"2024-09-11T05:04:17.106062+0000","last_fullsized":"2024-09-11T05:04:17.106062+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":963,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10215","reported_seq":10338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.166810+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:16.166810+0000","last_peered":"2024-09-11T05:04:16.166810+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:16.166810+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21072192,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10009","reported_seq":10121,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.240571+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:17.240571+0000","last_peered":"2024-09-11T05:04:17.240571+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:17.240571+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10007,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":213}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10068","reported_seq":10185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.502513+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:16.502513+0000","last_peered":"2024-09-11T05:04:16.502513+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:16.502513+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21171952,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10065,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9880","reported_seq":14481,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:16.751387+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:16.751387+0000","last_peered":"2024-09-11T05:04:16.751387+0000","last_clean":"2024-09-11T05:04:16.751387+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:16.751387+0000","last_undegraded":"2024-09-11T05:04:16.751387+0000","last_fullsized":"2024-09-11T05:04:16.751387+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20512768,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10150","reported_seq":11801,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.271033+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:17.271033+0000","last_peered":"2024-09-11T05:04:17.271033+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:17.271033+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21076560,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10149,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1023}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10052","reported_seq":10158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.466317+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:16.466317+0000","last_peered":"2024-09-11T05:04:16.466317+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:16.466317+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20804768,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":276}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9973","reported_seq":10092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.257203+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:17.257203+0000","last_peered":"2024-09-11T05:04:17.257203+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:17.257203+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19952800,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":133}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10144","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.753741+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:14.753741+0000","last_peered":"2024-09-11T05:04:14.753741+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:14.753741+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21566624,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":335581672,"num_objects":40970,"num_object_clones":0,"num_object_copies":163880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144933,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":670855168,"data_stored":670855168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159737,"ondisk_log_size":159737,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763764,"kb_used_data":164232,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93608076,"statfs":{"total":96636764160,"available":95854669824,"internally_reserved":0,"allocated":168173568,"data_stored":167984366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758812,"kb_used_data":163972,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93613028,"statfs":{"total":96636764160,"available":95859740672,"internally_reserved":0,"allocated":167907328,"data_stored":167718126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,3,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605540,"kb_used_data":164372,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93766300,"statfs":{"total":96636764160,"available":96016691200,"internally_reserved":0,"allocated":168316928,"data_stored":168127726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451735250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,6,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756596,"kb_used_data":165472,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93615244,"statfs":{"total":96636764160,"available":95862009856,"internally_reserved":0,"allocated":169443328,"data_stored":169254126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169086976,"data_stored":169086976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167194624,"data_stored":167194624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166965248,"data_stored":166965248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167608320,"data_stored":167608320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:19.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:19.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9907", 'reported_seq': 10001, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:16.388073+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:16.388073+0000', 'last_peered': '2024-09-11T05:04:16.388073+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:16.388073+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9907, 'log_dups_size': 0, 'ondisk_log_size': 9907, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21356544, 'num_objects': 2607, 'num_object_clones': 0, 'num_object_copies': 10428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9907, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10096", 'reported_seq': 10209, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:14.632128+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:14.632128+0000', 'last_peered': '2024-09-11T05:04:14.632128+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:14.632128+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21640352, 'num_objects': 2642, 'num_object_clones': 0, 'num_object_copies': 10568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10094, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9661", 'reported_seq': 9765, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:17.159139+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:17.159139+0000', 'last_peered': '2024-09-11T05:04:17.159139+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:17.159139+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9661, 'log_dups_size': 0, 'ondisk_log_size': 9661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20470376, 'num_objects': 2500, 'num_object_clones': 0, 'num_object_copies': 10000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2500, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9660, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 289}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9850", 'reported_seq': 9958, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:14.684833+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:14.684833+0000', 'last_peered': '2024-09-11T05:04:14.684833+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:14.684833+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9850, 'log_dups_size': 0, 'ondisk_log_size': 9850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21017760, 'num_objects': 2566, 'num_object_clones': 0, 'num_object_copies': 10264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9848, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 305}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9987", 'reported_seq': 10110, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:16.403456+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:16.403456+0000', 'last_peered': '2024-09-11T05:04:16.403456+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:16.403456+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9987, 'log_dups_size': 0, 'ondisk_log_size': 9987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20837536, 'num_objects': 2544, 'num_object_clones': 0, 'num_object_copies': 10176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9985, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 323}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:19.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10086", 'reported_seq': 10191, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:16.289335+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:16.289335+0000', 'last_peered': '2024-09-11T05:04:16.289335+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:16.289335+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20922368, 'num_objects': 2554, 'num_object_clones': 0, 'num_object_copies': 10216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10086, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 260}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10106", 'reported_seq': 10200, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:14.527224+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:14.527224+0000', 'last_peered': '2024-09-11T05:04:14.527224+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:14.527224+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21492896, 'num_objects': 2624, 'num_object_clones': 0, 'num_object_copies': 10496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10104, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 317}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10215", 'reported_seq': 10338, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:16.166810+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:16.166810+0000', 'last_peered': '2024-09-11T05:04:16.166810+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:16.166810+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21072192, 'num_objects': 2573, 'num_object_clones': 0, 'num_object_copies': 10292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10212, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 207}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10009", 'reported_seq': 10121, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:17.240571+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:17.240571+0000', 'last_peered': '2024-09-11T05:04:17.240571+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:17.240571+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20632736, 'num_objects': 2519, 'num_object_clones': 0, 'num_object_copies': 10076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10007, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 213}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10068", 'reported_seq': 10185, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:16.502513+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:16.502513+0000', 'last_peered': '2024-09-11T05:04:16.502513+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:16.502513+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21171952, 'num_objects': 2585, 'num_object_clones': 0, 'num_object_copies': 10340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10065, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 287}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:19.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10150", 'reported_seq': 11801, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:17.271033+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:17.271033+0000', 'last_peered': '2024-09-11T05:04:17.271033+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:17.271033+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21076560, 'num_objects': 2573, 'num_object_clones': 0, 'num_object_copies': 10292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1550, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10149, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 784, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1023}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:19.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:19.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10052", 'reported_seq': 10158, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:16.466317+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:16.466317+0000', 'last_peered': '2024-09-11T05:04:16.466317+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:16.466317+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20804768, 'num_objects': 2540, 'num_object_clones': 0, 'num_object_copies': 10160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10050, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 276}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:19.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:19.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9973", 'reported_seq': 10092, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:17.257203+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:17.257203+0000', 'last_peered': '2024-09-11T05:04:17.257203+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:17.257203+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9973, 'log_dups_size': 0, 'ondisk_log_size': 9973, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19952800, 'num_objects': 2436, 'num_object_clones': 0, 'num_object_copies': 9744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9971, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 133}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:19.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:19.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10144", 'reported_seq': 10263, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:14.753741+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:14.753741+0000', 'last_peered': '2024-09-11T05:04:14.753741+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:14.753741+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21566624, 'num_objects': 2633, 'num_object_clones': 0, 'num_object_copies': 10532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10142, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 345}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:19.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:19.913 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:19.913 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31241/163880 objects degraded (19.063%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":586,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40970,"data_bytes":335581672,"bytes_used":2953945088,"bytes_avail":383593111552,"bytes_total":386547056640,"degraded_objects":31241,"degraded_total":163880,"degraded_ratio":0.1906333902855748,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":227},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:19.932 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:19.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:20.572 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:20.572 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:20.603 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-11T05:04:19.830643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":335581672,"num_objects":40970,"num_object_clones":0,"num_object_copies":163880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144933,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159737,"ondisk_log_size":159737,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2884712,"kb_used_data":658048,"kb_used_omap":14,"kb_used_meta":2226545,"kb_avail":374602648,"statfs":{"total":386547056640,"available":383593111552,"internally_reserved":0,"allocated":673841152,"data_stored":673084344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2279982646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,11,11,4,12,12,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15998976,"num_objects":-1953,"num_object_clones":0,"num_object_copies":-7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005118"},"pg_stats":[{"pgid":"1.f","version":"35'9907","reported_seq":10001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.388073+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:16.388073+0000","last_peered":"2024-09-11T05:04:16.388073+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:16.388073+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9907,"log_dups_size":0,"ondisk_log_size":9907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21356544,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10096","reported_seq":10209,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.632128+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:14.632128+0000","last_peered":"2024-09-11T05:04:14.632128+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:14.632128+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21640352,"num_objects":2642,"num_object_clones":0,"num_object_copies":10568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10094,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9661","reported_seq":9765,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.159139+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:17.159139+0000","last_peered":"2024-09-11T05:04:17.159139+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:17.159139+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20470376,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9660,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":289}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9850","reported_seq":9958,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.684833+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:14.684833+0000","last_peered":"2024-09-11T05:04:14.684833+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:14.684833+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21017760,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":305}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9987","reported_seq":10110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.403456+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:16.403456+0000","last_peered":"2024-09-11T05:04:16.403456+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:16.403456+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9987,"log_dups_size":0,"ondisk_log_size":9987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20837536,"num_objects":2544,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9985,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10086","reported_seq":10191,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.289335+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:16.289335+0000","last_peered":"2024-09-11T05:04:16.289335+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:16.289335+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20922368,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10106","reported_seq":10200,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.527224+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:14.527224+0000","last_peered":"2024-09-11T05:04:14.527224+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:14.527224+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21492896,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10104,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":317}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10253","reported_seq":15110,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:17.106062+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:17.106062+0000","last_peered":"2024-09-11T05:04:17.106062+0000","last_clean":"2024-09-11T05:04:17.106062+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:17.106062+0000","last_undegraded":"2024-09-11T05:04:17.106062+0000","last_fullsized":"2024-09-11T05:04:17.106062+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":963,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10215","reported_seq":10338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.166810+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:16.166810+0000","last_peered":"2024-09-11T05:04:16.166810+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:16.166810+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21072192,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10009","reported_seq":10121,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.240571+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:17.240571+0000","last_peered":"2024-09-11T05:04:17.240571+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:17.240571+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632736,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10007,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":213}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10068","reported_seq":10185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.502513+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:16.502513+0000","last_peered":"2024-09-11T05:04:16.502513+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:16.502513+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21171952,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10065,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9880","reported_seq":14481,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:16.751387+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:16.751387+0000","last_peered":"2024-09-11T05:04:16.751387+0000","last_clean":"2024-09-11T05:04:16.751387+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:16.751387+0000","last_undegraded":"2024-09-11T05:04:16.751387+0000","last_fullsized":"2024-09-11T05:04:16.751387+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20512768,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10150","reported_seq":11801,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.271033+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:17.271033+0000","last_peered":"2024-09-11T05:04:17.271033+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:17.271033+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21076560,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10149,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1023}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10052","reported_seq":10158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:16.466317+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:16.466317+0000","last_peered":"2024-09-11T05:04:16.466317+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:16.466317+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20804768,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":276}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9973","reported_seq":10092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:17.257203+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:17.257203+0000","last_peered":"2024-09-11T05:04:17.257203+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:17.257203+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19952800,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":133}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10144","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:14.753741+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:14.753741+0000","last_peered":"2024-09-11T05:04:14.753741+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:14.753741+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21566624,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":335581672,"num_objects":40970,"num_object_clones":0,"num_object_copies":163880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144933,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":670855168,"data_stored":670855168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159737,"ondisk_log_size":159737,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763764,"kb_used_data":164232,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93608076,"statfs":{"total":96636764160,"available":95854669824,"internally_reserved":0,"allocated":168173568,"data_stored":167984366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,0,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758812,"kb_used_data":163972,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93613028,"statfs":{"total":96636764160,"available":95859740672,"internally_reserved":0,"allocated":167907328,"data_stored":167718126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,3,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605540,"kb_used_data":164372,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93766300,"statfs":{"total":96636764160,"available":96016691200,"internally_reserved":0,"allocated":168316928,"data_stored":168127726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":451735250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,6,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756596,"kb_used_data":165472,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93615244,"statfs":{"total":96636764160,"available":95862009856,"internally_reserved":0,"allocated":169443328,"data_stored":169254126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169086976,"data_stored":169086976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167194624,"data_stored":167194624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166965248,"data_stored":166965248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167608320,"data_stored":167608320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:23.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:24.249 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:24.273 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-11T05:04:23.834395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":327520744,"num_objects":39986,"num_object_clones":0,"num_object_copies":159944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145917,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160021,"ondisk_log_size":160021,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868440,"kb_used_data":641804,"kb_used_omap":14,"kb_used_meta":2226545,"kb_avail":374618920,"statfs":{"total":386547056640,"available":383609774080,"internally_reserved":0,"allocated":657207296,"data_stored":656450488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2279982646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,11,8,7,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18931712,"num_objects":-2311,"num_object_clones":0,"num_object_copies":-9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2311,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007138"},"pg_stats":[{"pgid":"1.f","version":"35'9967","reported_seq":10061,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.426875+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:21.426875+0000","last_peered":"2024-09-11T05:04:21.426875+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:21.426875+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20865024,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10149","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:19.823496+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:19.823496+0000","last_peered":"2024-09-11T05:04:19.823496+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:19.823496+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21206176,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9721","reported_seq":9825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.298812+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:22.298812+0000","last_peered":"2024-09-11T05:04:22.298812+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:22.298812+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9721,"log_dups_size":0,"ondisk_log_size":9721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19978856,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9720,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":229}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9908","reported_seq":10016,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:19.270144+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:19.270144+0000","last_peered":"2024-09-11T05:04:19.270144+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:19.270144+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9908,"log_dups_size":0,"ondisk_log_size":9908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20542624,"num_objects":2508,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10072","reported_seq":10195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.523126+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:21.523126+0000","last_peered":"2024-09-11T05:04:21.523126+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:21.523126+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141216,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10070,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10140","reported_seq":10244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.536075+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:21.536075+0000","last_peered":"2024-09-11T05:04:21.536075+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:21.536075+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20480000,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10140,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10165","reported_seq":10259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:19.694807+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:19.694807+0000","last_peered":"2024-09-11T05:04:19.694807+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:19.694807+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21009568,"num_objects":2565,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10163,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":258}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10324","reported_seq":15181,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:22.120932+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:22.120932+0000","last_peered":"2024-09-11T05:04:22.120932+0000","last_clean":"2024-09-11T05:04:22.120932+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:22.120932+0000","last_undegraded":"2024-09-11T05:04:22.120932+0000","last_fullsized":"2024-09-11T05:04:22.120932+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20471808,"num_objects":2499,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10274","reported_seq":10397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.318073+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:21.318073+0000","last_peered":"2024-09-11T05:04:21.318073+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:21.318073+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20588864,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10271,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10078","reported_seq":10190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.187742+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:22.187742+0000","last_peered":"2024-09-11T05:04:22.187742+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:22.187742+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067488,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":144}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10119","reported_seq":10235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.427108+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:21.427108+0000","last_peered":"2024-09-11T05:04:21.427108+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:21.427108+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20754160,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10116,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9927","reported_seq":14528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:22.124425+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:22.124425+0000","last_peered":"2024-09-11T05:04:22.124425+0000","last_clean":"2024-09-11T05:04:22.124425+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:22.124425+0000","last_undegraded":"2024-09-11T05:04:22.124425+0000","last_fullsized":"2024-09-11T05:04:22.124425+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20127744,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10209","reported_seq":11971,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.362917+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:22.362917+0000","last_peered":"2024-09-11T05:04:22.362917+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:22.362917+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20593232,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":839,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10117","reported_seq":10224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.573818+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:21.573818+0000","last_peered":"2024-09-11T05:04:21.573818+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:21.573818+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20272288,"num_objects":2475,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10115,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":211}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10043","reported_seq":10160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.370985+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:22.370985+0000","last_peered":"2024-09-11T05:04:22.370985+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:22.370985+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379360,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10041,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":63}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10208","reported_seq":10326,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:19.707986+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:19.707986+0000","last_peered":"2024-09-11T05:04:19.707986+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:19.707986+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21042336,"num_objects":2569,"num_object_clones":0,"num_object_copies":10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10206,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":327520744,"num_objects":39986,"num_object_clones":0,"num_object_copies":159944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145917,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":654831616,"data_stored":654831616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160021,"ondisk_log_size":160021,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764092,"kb_used_data":160424,"kb_used_omap":4,"kb_used_meta":603643,"kb_avail":93607748,"statfs":{"total":96636764160,"available":95854333952,"internally_reserved":0,"allocated":164274176,"data_stored":164084974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":618131152},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759132,"kb_used_data":160132,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93612708,"statfs":{"total":96636764160,"available":95859412992,"internally_reserved":0,"allocated":163975168,"data_stored":163785966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588428,"kb_used_data":159740,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93783412,"statfs":{"total":96636764160,"available":96034213888,"internally_reserved":0,"allocated":163573760,"data_stored":163384558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":438955730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756788,"kb_used_data":161508,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93615052,"statfs":{"total":96636764160,"available":95861813248,"internally_reserved":0,"allocated":165384192,"data_stored":165194990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165015552,"data_stored":165015552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163213312,"data_stored":163213312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162963456,"data_stored":162963456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":163639296,"data_stored":163639296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:24.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:24.901 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:24.903 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:24.931 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-11T05:04:23.834395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":327520744,"num_objects":39986,"num_object_clones":0,"num_object_copies":159944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145917,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160021,"ondisk_log_size":160021,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868440,"kb_used_data":641804,"kb_used_omap":14,"kb_used_meta":2226545,"kb_avail":374618920,"statfs":{"total":386547056640,"available":383609774080,"internally_reserved":0,"allocated":657207296,"data_stored":656450488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2279982646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,11,8,7,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18931712,"num_objects":-2311,"num_object_clones":0,"num_object_copies":-9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2311,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007138"},"pg_stats":[{"pgid":"1.f","version":"35'9967","reported_seq":10061,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.426875+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:21.426875+0000","last_peered":"2024-09-11T05:04:21.426875+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:21.426875+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20865024,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10149","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:19.823496+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:19.823496+0000","last_peered":"2024-09-11T05:04:19.823496+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:19.823496+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21206176,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9721","reported_seq":9825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.298812+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:22.298812+0000","last_peered":"2024-09-11T05:04:22.298812+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:22.298812+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9721,"log_dups_size":0,"ondisk_log_size":9721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19978856,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9720,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":229}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9908","reported_seq":10016,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:19.270144+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:19.270144+0000","last_peered":"2024-09-11T05:04:19.270144+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:19.270144+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9908,"log_dups_size":0,"ondisk_log_size":9908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20542624,"num_objects":2508,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10072","reported_seq":10195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.523126+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:21.523126+0000","last_peered":"2024-09-11T05:04:21.523126+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:21.523126+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141216,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10070,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10140","reported_seq":10244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.536075+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:21.536075+0000","last_peered":"2024-09-11T05:04:21.536075+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:21.536075+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20480000,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10140,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10165","reported_seq":10259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:19.694807+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:19.694807+0000","last_peered":"2024-09-11T05:04:19.694807+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:19.694807+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21009568,"num_objects":2565,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10163,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":258}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10324","reported_seq":15181,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:22.120932+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:22.120932+0000","last_peered":"2024-09-11T05:04:22.120932+0000","last_clean":"2024-09-11T05:04:22.120932+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:22.120932+0000","last_undegraded":"2024-09-11T05:04:22.120932+0000","last_fullsized":"2024-09-11T05:04:22.120932+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20471808,"num_objects":2499,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10274","reported_seq":10397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.318073+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:21.318073+0000","last_peered":"2024-09-11T05:04:21.318073+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:21.318073+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20588864,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10271,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10078","reported_seq":10190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.187742+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:22.187742+0000","last_peered":"2024-09-11T05:04:22.187742+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:22.187742+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067488,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":144}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10119","reported_seq":10235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.427108+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:21.427108+0000","last_peered":"2024-09-11T05:04:21.427108+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:21.427108+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20754160,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10116,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9927","reported_seq":14528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:22.124425+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:22.124425+0000","last_peered":"2024-09-11T05:04:22.124425+0000","last_clean":"2024-09-11T05:04:22.124425+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:22.124425+0000","last_undegraded":"2024-09-11T05:04:22.124425+0000","last_fullsized":"2024-09-11T05:04:22.124425+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20127744,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10209","reported_seq":11971,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.362917+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:22.362917+0000","last_peered":"2024-09-11T05:04:22.362917+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:22.362917+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20593232,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":839,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10117","reported_seq":10224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.573818+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:21.573818+0000","last_peered":"2024-09-11T05:04:21.573818+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:21.573818+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20272288,"num_objects":2475,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10115,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":211}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10043","reported_seq":10160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.370985+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:22.370985+0000","last_peered":"2024-09-11T05:04:22.370985+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:22.370985+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379360,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10041,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":63}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10208","reported_seq":10326,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:19.707986+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:19.707986+0000","last_peered":"2024-09-11T05:04:19.707986+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:19.707986+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21042336,"num_objects":2569,"num_object_clones":0,"num_object_copies":10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10206,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":327520744,"num_objects":39986,"num_object_clones":0,"num_object_copies":159944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145917,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":654831616,"data_stored":654831616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160021,"ondisk_log_size":160021,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764092,"kb_used_data":160424,"kb_used_omap":4,"kb_used_meta":603643,"kb_avail":93607748,"statfs":{"total":96636764160,"available":95854333952,"internally_reserved":0,"allocated":164274176,"data_stored":164084974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":618131152},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759132,"kb_used_data":160132,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93612708,"statfs":{"total":96636764160,"available":95859412992,"internally_reserved":0,"allocated":163975168,"data_stored":163785966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588428,"kb_used_data":159740,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93783412,"statfs":{"total":96636764160,"available":96034213888,"internally_reserved":0,"allocated":163573760,"data_stored":163384558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":438955730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756788,"kb_used_data":161508,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93615052,"statfs":{"total":96636764160,"available":95861813248,"internally_reserved":0,"allocated":165384192,"data_stored":165194990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165015552,"data_stored":165015552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163213312,"data_stored":163213312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162963456,"data_stored":162963456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":163639296,"data_stored":163639296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:24.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:24.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9967", 'reported_seq': 10061, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:21.426875+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:21.426875+0000', 'last_peered': '2024-09-11T05:04:21.426875+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:21.426875+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9967, 'log_dups_size': 0, 'ondisk_log_size': 9967, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20865024, 'num_objects': 2547, 'num_object_clones': 0, 'num_object_copies': 10188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9967, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 256}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:24.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:24.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10149", 'reported_seq': 10263, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:19.823496+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:19.823496+0000', 'last_peered': '2024-09-11T05:04:19.823496+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:19.823496+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21206176, 'num_objects': 2589, 'num_object_clones': 0, 'num_object_copies': 10356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10147, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 331}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:24.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:24.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9721", 'reported_seq': 9825, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:22.298812+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:22.298812+0000', 'last_peered': '2024-09-11T05:04:22.298812+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:22.298812+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9721, 'log_dups_size': 0, 'ondisk_log_size': 9721, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19978856, 'num_objects': 2440, 'num_object_clones': 0, 'num_object_copies': 9760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2440, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9720, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 229}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:24.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:24.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9908", 'reported_seq': 10016, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:19.270144+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:19.270144+0000', 'last_peered': '2024-09-11T05:04:19.270144+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:19.270144+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9908, 'log_dups_size': 0, 'ondisk_log_size': 9908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20542624, 'num_objects': 2508, 'num_object_clones': 0, 'num_object_copies': 10032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9906, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 247}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:24.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:24.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10072", 'reported_seq': 10195, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:21.523126+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:21.523126+0000', 'last_peered': '2024-09-11T05:04:21.523126+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:21.523126+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20141216, 'num_objects': 2459, 'num_object_clones': 0, 'num_object_copies': 9836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10070, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 238}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:24.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:24.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10140", 'reported_seq': 10244, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:21.536075+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:21.536075+0000', 'last_peered': '2024-09-11T05:04:21.536075+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:21.536075+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20480000, 'num_objects': 2500, 'num_object_clones': 0, 'num_object_copies': 10000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10140, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 206}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:24.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:24.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10165", 'reported_seq': 10259, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:19.694807+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:19.694807+0000', 'last_peered': '2024-09-11T05:04:19.694807+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:19.694807+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21009568, 'num_objects': 2565, 'num_object_clones': 0, 'num_object_copies': 10260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10163, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 258}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:24.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:24.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10274", 'reported_seq': 10397, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:21.318073+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:21.318073+0000', 'last_peered': '2024-09-11T05:04:21.318073+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:21.318073+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20588864, 'num_objects': 2514, 'num_object_clones': 0, 'num_object_copies': 10056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10271, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 148}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:24.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10078", 'reported_seq': 10190, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:22.187742+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:22.187742+0000', 'last_peered': '2024-09-11T05:04:22.187742+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:22.187742+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20067488, 'num_objects': 2450, 'num_object_clones': 0, 'num_object_copies': 9800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10076, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 144}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10119", 'reported_seq': 10235, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:21.427108+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:21.427108+0000', 'last_peered': '2024-09-11T05:04:21.427108+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:21.427108+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20754160, 'num_objects': 2534, 'num_object_clones': 0, 'num_object_copies': 10136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10116, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 236}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10209", 'reported_seq': 11971, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:22.362917+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:22.362917+0000', 'last_peered': '2024-09-11T05:04:22.362917+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:22.362917+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20593232, 'num_objects': 2514, 'num_object_clones': 0, 'num_object_copies': 10056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1495, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10208, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 839, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10117", 'reported_seq': 10224, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:21.573818+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:21.573818+0000', 'last_peered': '2024-09-11T05:04:21.573818+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:21.573818+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20272288, 'num_objects': 2475, 'num_object_clones': 0, 'num_object_copies': 9900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10115, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 211}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10043", 'reported_seq': 10160, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:22.370985+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:22.370985+0000', 'last_peered': '2024-09-11T05:04:22.370985+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:22.370985+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19379360, 'num_objects': 2366, 'num_object_clones': 0, 'num_object_copies': 9464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10041, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 63}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:24.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10208", 'reported_seq': 10326, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:19.707986+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:19.707986+0000', 'last_peered': '2024-09-11T05:04:19.707986+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:19.707986+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21042336, 'num_objects': 2569, 'num_object_clones': 0, 'num_object_copies': 10276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10206, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:24.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:25.764 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:25.764 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31186/159944 objects degraded (19.498%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":592,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39986,"data_bytes":327520744,"bytes_used":2937282560,"bytes_avail":383609774080,"bytes_total":386547056640,"degraded_objects":31186,"degraded_total":159944,"degraded_ratio":0.19498074326014106,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":192},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:25.802 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:25.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:26.438 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:26.438 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:26.467 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-11T05:04:25.835227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":325595624,"num_objects":39751,"num_object_clones":0,"num_object_copies":159004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39751,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146152,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160156,"ondisk_log_size":160156,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868760,"kb_used_data":637964,"kb_used_omap":14,"kb_used_meta":2230705,"kb_avail":374618600,"statfs":{"total":386547056640,"available":383609446400,"internally_reserved":0,"allocated":653275136,"data_stored":652518328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2284242486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,8,6,6,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18112512,"num_objects":-2211,"num_object_clones":0,"num_object_copies":-8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007104"},"pg_stats":[{"pgid":"1.f","version":"35'9967","reported_seq":10061,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.426875+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:21.426875+0000","last_peered":"2024-09-11T05:04:21.426875+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:21.426875+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20865024,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10200","reported_seq":10314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.520185+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:24.520185+0000","last_peered":"2024-09-11T05:04:24.520185+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:24.520185+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20788384,"num_objects":2538,"num_object_clones":0,"num_object_copies":10152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9721","reported_seq":9825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.298812+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:22.298812+0000","last_peered":"2024-09-11T05:04:22.298812+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:22.298812+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9721,"log_dups_size":0,"ondisk_log_size":9721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19978856,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9720,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":229}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9973","reported_seq":10081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.566427+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:24.566427+0000","last_peered":"2024-09-11T05:04:24.566427+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:24.566427+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010144,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10072","reported_seq":10195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.523126+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:21.523126+0000","last_peered":"2024-09-11T05:04:21.523126+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:21.523126+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141216,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10070,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10140","reported_seq":10244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.536075+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:21.536075+0000","last_peered":"2024-09-11T05:04:21.536075+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:21.536075+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20480000,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10140,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10228","reported_seq":10322,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.799444+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:24.799444+0000","last_peered":"2024-09-11T05:04:24.799444+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:24.799444+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493472,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":195}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10324","reported_seq":15181,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:22.120932+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:22.120932+0000","last_peered":"2024-09-11T05:04:22.120932+0000","last_clean":"2024-09-11T05:04:22.120932+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:22.120932+0000","last_undegraded":"2024-09-11T05:04:22.120932+0000","last_fullsized":"2024-09-11T05:04:22.120932+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20471808,"num_objects":2499,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10274","reported_seq":10397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.318073+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:21.318073+0000","last_peered":"2024-09-11T05:04:21.318073+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:21.318073+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20588864,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10271,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10078","reported_seq":10190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.187742+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:22.187742+0000","last_peered":"2024-09-11T05:04:22.187742+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:22.187742+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067488,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":144}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10119","reported_seq":10235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.427108+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:21.427108+0000","last_peered":"2024-09-11T05:04:21.427108+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:21.427108+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20754160,"num_objects":2534,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10116,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9927","reported_seq":14528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:22.124425+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:22.124425+0000","last_peered":"2024-09-11T05:04:22.124425+0000","last_clean":"2024-09-11T05:04:22.124425+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:22.124425+0000","last_undegraded":"2024-09-11T05:04:22.124425+0000","last_fullsized":"2024-09-11T05:04:22.124425+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20127744,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10209","reported_seq":11971,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.362917+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:22.362917+0000","last_peered":"2024-09-11T05:04:22.362917+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:22.362917+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20593232,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":839,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10117","reported_seq":10224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:21.573818+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:21.573818+0000","last_peered":"2024-09-11T05:04:21.573818+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:21.573818+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20272288,"num_objects":2475,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10115,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":211}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10043","reported_seq":10160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:22.370985+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:22.370985+0000","last_peered":"2024-09-11T05:04:22.370985+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:22.370985+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379360,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10041,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":63}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10264","reported_seq":10384,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.787609+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:24.787609+0000","last_peered":"2024-09-11T05:04:24.787609+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:24.787609+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20583584,"num_objects":2513,"num_object_clones":0,"num_object_copies":10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10262,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":325595624,"num_objects":39751,"num_object_clones":0,"num_object_copies":159004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39751,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146152,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":650833920,"data_stored":650833920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160156,"ondisk_log_size":160156,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764092,"kb_used_data":160424,"kb_used_omap":4,"kb_used_meta":603643,"kb_avail":93607748,"statfs":{"total":96636764160,"available":95854333952,"internally_reserved":0,"allocated":164274176,"data_stored":164084974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":618131152},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759132,"kb_used_data":160132,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93612708,"statfs":{"total":96636764160,"available":95859412992,"internally_reserved":0,"allocated":163975168,"data_stored":163785966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588428,"kb_used_data":159740,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93783412,"statfs":{"total":96636764160,"available":96034213888,"internally_reserved":0,"allocated":163573760,"data_stored":163384558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":438955730},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757108,"kb_used_data":157668,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93614732,"statfs":{"total":96636764160,"available":95861485568,"internally_reserved":0,"allocated":161452032,"data_stored":161262830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161017856,"data_stored":161017856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163213312,"data_stored":163213312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162963456,"data_stored":162963456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":163639296,"data_stored":163639296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:29.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:30.079 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:30.079 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:30.101 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-11T05:04:29.836597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":319713768,"num_objects":39033,"num_object_clones":0,"num_object_copies":156132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146870,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160574,"ondisk_log_size":160574,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2854536,"kb_used_data":626500,"kb_used_omap":14,"kb_used_meta":2227953,"kb_avail":374632824,"statfs":{"total":386547056640,"available":383624011776,"internally_reserved":0,"allocated":641536000,"data_stored":640779192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2281424438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,1,4,1,13,14,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15867904,"num_objects":-1937,"num_object_clones":0,"num_object_copies":-7748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1937,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006646"},"pg_stats":[{"pgid":"1.f","version":"35'10028","reported_seq":10122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.500124+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:26.500124+0000","last_peered":"2024-09-11T05:04:26.500124+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:26.500124+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20365312,"num_objects":2486,"num_object_clones":0,"num_object_copies":9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10200","reported_seq":10314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.520185+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:24.520185+0000","last_peered":"2024-09-11T05:04:24.520185+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:24.520185+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20788384,"num_objects":2538,"num_object_clones":0,"num_object_copies":10152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9780","reported_seq":9885,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.217462+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:27.217462+0000","last_peered":"2024-09-11T05:04:27.217462+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:27.217462+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9780,"log_dups_size":0,"ondisk_log_size":9780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19495528,"num_objects":2381,"num_object_clones":0,"num_object_copies":9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9779,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":170}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9973","reported_seq":10081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.566427+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:24.566427+0000","last_peered":"2024-09-11T05:04:24.566427+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:24.566427+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010144,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10142","reported_seq":10265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.576570+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:26.576570+0000","last_peered":"2024-09-11T05:04:26.576570+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:26.576570+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19567776,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10140,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10198","reported_seq":10301,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.626249+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:26.626249+0000","last_peered":"2024-09-11T05:04:26.626249+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:26.626249+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20004864,"num_objects":2442,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10228","reported_seq":10322,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.799444+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:24.799444+0000","last_peered":"2024-09-11T05:04:24.799444+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:24.799444+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493472,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":195}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10389","reported_seq":15246,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:26.955177+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:26.955177+0000","last_peered":"2024-09-11T05:04:26.955177+0000","last_clean":"2024-09-11T05:04:26.955177+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:26.955177+0000","last_undegraded":"2024-09-11T05:04:26.955177+0000","last_fullsized":"2024-09-11T05:04:26.955177+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19939328,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10325","reported_seq":10448,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.502858+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:26.502858+0000","last_peered":"2024-09-11T05:04:26.502858+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:26.502858+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20171072,"num_objects":2463,"num_object_clones":0,"num_object_copies":9852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":97}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10131","reported_seq":10241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.375128+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:27.375128+0000","last_peered":"2024-09-11T05:04:27.375128+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:27.375128+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633312,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10129,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":91}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10186","reported_seq":10303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.593220+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:26.593220+0000","last_peered":"2024-09-11T05:04:26.593220+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:26.593220+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20205296,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9977","reported_seq":14578,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:27.212439+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:27.212439+0000","last_peered":"2024-09-11T05:04:27.212439+0000","last_clean":"2024-09-11T05:04:27.212439+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:27.212439+0000","last_undegraded":"2024-09-11T05:04:27.212439+0000","last_fullsized":"2024-09-11T05:04:27.212439+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9977,"log_dups_size":0,"ondisk_log_size":9977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19718144,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3787,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10274","reported_seq":12132,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.480056+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:27.480056+0000","last_peered":"2024-09-11T05:04:27.480056+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:27.480056+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060752,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10179","reported_seq":10286,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.440138+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:26.440138+0000","last_peered":"2024-09-11T05:04:26.440138+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:26.440138+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19764384,"num_objects":2413,"num_object_clones":0,"num_object_copies":9652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10177,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":149}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10100","reported_seq":10219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.116210+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:27.116210+0000","last_peered":"2024-09-11T05:04:27.116210+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:27.116210+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18912416,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10098,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10264","reported_seq":10384,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.787609+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:24.787609+0000","last_peered":"2024-09-11T05:04:24.787609+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:24.787609+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20583584,"num_objects":2513,"num_object_clones":0,"num_object_copies":10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10262,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":319713768,"num_objects":39033,"num_object_clones":0,"num_object_copies":156132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146870,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638955520,"data_stored":638955520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160574,"ondisk_log_size":160574,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747760,"kb_used_data":156576,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93624080,"statfs":{"total":96636764160,"available":95871057920,"internally_reserved":0,"allocated":160333824,"data_stored":160144622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742884,"kb_used_data":156372,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93628956,"statfs":{"total":96636764160,"available":95876050944,"internally_reserved":0,"allocated":160124928,"data_stored":159935726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,0,0,6,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606784,"kb_used_data":155884,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93765056,"statfs":{"total":96636764160,"available":96015417344,"internally_reserved":0,"allocated":159625216,"data_stored":159436014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461696722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757108,"kb_used_data":157668,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93614732,"statfs":{"total":96636764160,"available":95861485568,"internally_reserved":0,"allocated":161452032,"data_stored":161262830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161017856,"data_stored":161017856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159256576,"data_stored":159256576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":158978048,"data_stored":158978048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159703040,"data_stored":159703040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:30.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:30.736 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:30.736 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:30.757 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-11T05:04:29.836597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":319713768,"num_objects":39033,"num_object_clones":0,"num_object_copies":156132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146870,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160574,"ondisk_log_size":160574,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2854536,"kb_used_data":626500,"kb_used_omap":14,"kb_used_meta":2227953,"kb_avail":374632824,"statfs":{"total":386547056640,"available":383624011776,"internally_reserved":0,"allocated":641536000,"data_stored":640779192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2281424438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,1,4,1,13,14,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15867904,"num_objects":-1937,"num_object_clones":0,"num_object_copies":-7748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1937,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006646"},"pg_stats":[{"pgid":"1.f","version":"35'10028","reported_seq":10122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.500124+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:26.500124+0000","last_peered":"2024-09-11T05:04:26.500124+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:26.500124+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20365312,"num_objects":2486,"num_object_clones":0,"num_object_copies":9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10200","reported_seq":10314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.520185+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:24.520185+0000","last_peered":"2024-09-11T05:04:24.520185+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:24.520185+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20788384,"num_objects":2538,"num_object_clones":0,"num_object_copies":10152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9780","reported_seq":9885,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.217462+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:27.217462+0000","last_peered":"2024-09-11T05:04:27.217462+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:27.217462+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9780,"log_dups_size":0,"ondisk_log_size":9780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19495528,"num_objects":2381,"num_object_clones":0,"num_object_copies":9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9779,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":170}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9973","reported_seq":10081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.566427+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:24.566427+0000","last_peered":"2024-09-11T05:04:24.566427+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:24.566427+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010144,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10142","reported_seq":10265,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.576570+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:26.576570+0000","last_peered":"2024-09-11T05:04:26.576570+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:26.576570+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19567776,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10140,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10198","reported_seq":10301,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.626249+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:26.626249+0000","last_peered":"2024-09-11T05:04:26.626249+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:26.626249+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20004864,"num_objects":2442,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10228","reported_seq":10322,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.799444+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:24.799444+0000","last_peered":"2024-09-11T05:04:24.799444+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:24.799444+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493472,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":195}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10389","reported_seq":15246,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:26.955177+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:26.955177+0000","last_peered":"2024-09-11T05:04:26.955177+0000","last_clean":"2024-09-11T05:04:26.955177+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:26.955177+0000","last_undegraded":"2024-09-11T05:04:26.955177+0000","last_fullsized":"2024-09-11T05:04:26.955177+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19939328,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10325","reported_seq":10448,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.502858+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:26.502858+0000","last_peered":"2024-09-11T05:04:26.502858+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:26.502858+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20171072,"num_objects":2463,"num_object_clones":0,"num_object_copies":9852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":97}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10131","reported_seq":10241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.375128+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:27.375128+0000","last_peered":"2024-09-11T05:04:27.375128+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:27.375128+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633312,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10129,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":91}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10186","reported_seq":10303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.593220+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:26.593220+0000","last_peered":"2024-09-11T05:04:26.593220+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:26.593220+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20205296,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9977","reported_seq":14578,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:27.212439+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:27.212439+0000","last_peered":"2024-09-11T05:04:27.212439+0000","last_clean":"2024-09-11T05:04:27.212439+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:27.212439+0000","last_undegraded":"2024-09-11T05:04:27.212439+0000","last_fullsized":"2024-09-11T05:04:27.212439+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9977,"log_dups_size":0,"ondisk_log_size":9977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19718144,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3787,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10274","reported_seq":12132,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.480056+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:27.480056+0000","last_peered":"2024-09-11T05:04:27.480056+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:27.480056+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060752,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10179","reported_seq":10286,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:26.440138+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:26.440138+0000","last_peered":"2024-09-11T05:04:26.440138+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:26.440138+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19764384,"num_objects":2413,"num_object_clones":0,"num_object_copies":9652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10177,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":149}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10100","reported_seq":10219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.116210+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:27.116210+0000","last_peered":"2024-09-11T05:04:27.116210+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:27.116210+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18912416,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10098,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10264","reported_seq":10384,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:24.787609+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:24.787609+0000","last_peered":"2024-09-11T05:04:24.787609+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:24.787609+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20583584,"num_objects":2513,"num_object_clones":0,"num_object_copies":10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10262,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":319713768,"num_objects":39033,"num_object_clones":0,"num_object_copies":156132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146870,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638955520,"data_stored":638955520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160574,"ondisk_log_size":160574,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747760,"kb_used_data":156576,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93624080,"statfs":{"total":96636764160,"available":95871057920,"internally_reserved":0,"allocated":160333824,"data_stored":160144622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742884,"kb_used_data":156372,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93628956,"statfs":{"total":96636764160,"available":95876050944,"internally_reserved":0,"allocated":160124928,"data_stored":159935726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,0,0,6,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606784,"kb_used_data":155884,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93765056,"statfs":{"total":96636764160,"available":96015417344,"internally_reserved":0,"allocated":159625216,"data_stored":159436014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461696722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757108,"kb_used_data":157668,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93614732,"statfs":{"total":96636764160,"available":95861485568,"internally_reserved":0,"allocated":161452032,"data_stored":161262830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161017856,"data_stored":161017856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159256576,"data_stored":159256576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":158978048,"data_stored":158978048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159703040,"data_stored":159703040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:30.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:30.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10028", 'reported_seq': 10122, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:26.500124+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:26.500124+0000', 'last_peered': '2024-09-11T05:04:26.500124+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:26.500124+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20365312, 'num_objects': 2486, 'num_object_clones': 0, 'num_object_copies': 9944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10028, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 195}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:30.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:30.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10200", 'reported_seq': 10314, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:24.520185+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:24.520185+0000', 'last_peered': '2024-09-11T05:04:24.520185+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:24.520185+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20788384, 'num_objects': 2538, 'num_object_clones': 0, 'num_object_copies': 10152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10198, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:30.759 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:30.759 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9780", 'reported_seq': 9885, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:27.217462+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:27.217462+0000', 'last_peered': '2024-09-11T05:04:27.217462+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:27.217462+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9780, 'log_dups_size': 0, 'ondisk_log_size': 9780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19495528, 'num_objects': 2381, 'num_object_clones': 0, 'num_object_copies': 9524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2381, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9779, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 170}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:30.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:30.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9973", 'reported_seq': 10081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:24.566427+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:24.566427+0000', 'last_peered': '2024-09-11T05:04:24.566427+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:24.566427+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9973, 'log_dups_size': 0, 'ondisk_log_size': 9973, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20010144, 'num_objects': 2443, 'num_object_clones': 0, 'num_object_copies': 9772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9971, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 182}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:30.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:30.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10142", 'reported_seq': 10265, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:26.576570+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:26.576570+0000', 'last_peered': '2024-09-11T05:04:26.576570+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:26.576570+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19567776, 'num_objects': 2389, 'num_object_clones': 0, 'num_object_copies': 9556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10140, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 168}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:30.760 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:30.760 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10198", 'reported_seq': 10301, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:26.626249+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:26.626249+0000', 'last_peered': '2024-09-11T05:04:26.626249+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:26.626249+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20004864, 'num_objects': 2442, 'num_object_clones': 0, 'num_object_copies': 9768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10198, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 148}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:30.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:30.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10228", 'reported_seq': 10322, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:24.799444+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:24.799444+0000', 'last_peered': '2024-09-11T05:04:24.799444+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:24.799444+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20493472, 'num_objects': 2502, 'num_object_clones': 0, 'num_object_copies': 10008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10226, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 195}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:30.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:30.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10325", 'reported_seq': 10448, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:26.502858+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:26.502858+0000', 'last_peered': '2024-09-11T05:04:26.502858+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:26.502858+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20171072, 'num_objects': 2463, 'num_object_clones': 0, 'num_object_copies': 9852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10322, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 97}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:30.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:30.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10131", 'reported_seq': 10241, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:27.375128+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:27.375128+0000', 'last_peered': '2024-09-11T05:04:27.375128+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:27.375128+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19633312, 'num_objects': 2397, 'num_object_clones': 0, 'num_object_copies': 9588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10129, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 91}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:30.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:30.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10186", 'reported_seq': 10303, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:26.593220+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:26.593220+0000', 'last_peered': '2024-09-11T05:04:26.593220+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:26.593220+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20205296, 'num_objects': 2467, 'num_object_clones': 0, 'num_object_copies': 9868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10183, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 169}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:30.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:30.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10274", 'reported_seq': 12132, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:27.480056+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:27.480056+0000', 'last_peered': '2024-09-11T05:04:27.480056+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:27.480056+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20060752, 'num_objects': 2449, 'num_object_clones': 0, 'num_object_copies': 9796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10273, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 888, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1003}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:30.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:30.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10179", 'reported_seq': 10286, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:26.440138+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:26.440138+0000', 'last_peered': '2024-09-11T05:04:26.440138+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:26.440138+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19764384, 'num_objects': 2413, 'num_object_clones': 0, 'num_object_copies': 9652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10177, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:30.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:30.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10100", 'reported_seq': 10219, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:27.116210+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:27.116210+0000', 'last_peered': '2024-09-11T05:04:27.116210+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:27.116210+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18912416, 'num_objects': 2309, 'num_object_clones': 0, 'num_object_copies': 9236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10098, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 6}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:30.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:30.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10264", 'reported_seq': 10384, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:24.787609+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:24.787609+0000', 'last_peered': '2024-09-11T05:04:24.787609+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:24.787609+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20583584, 'num_objects': 2513, 'num_object_clones': 0, 'num_object_copies': 10052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10262, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 225}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:30.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:31.545 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31137/156132 objects degraded (19.943%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":598,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39033,"data_bytes":319713768,"bytes_used":2923044864,"bytes_avail":383624011776,"bytes_total":386547056640,"degraded_objects":31137,"degraded_total":156132,"degraded_ratio":0.19942740757820307,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":161},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:31.580 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:31.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:32.234 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:32.235 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:32.261 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-11T05:04:31.838181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":314683880,"num_objects":38419,"num_object_clones":0,"num_object_copies":153676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147484,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160688,"ondisk_log_size":160688,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838264,"kb_used_data":618548,"kb_used_omap":14,"kb_used_meta":2219633,"kb_avail":374649096,"statfs":{"total":386547056640,"available":383640674304,"internally_reserved":0,"allocated":633393152,"data_stored":632636344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2272904758},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,0,2,13,23,22,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20897792,"num_objects":-2551,"num_object_clones":0,"num_object_copies":-10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2551,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007537"},"pg_stats":[{"pgid":"1.f","version":"35'10095","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.550607+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:31.550607+0000","last_peered":"2024-09-11T05:04:31.550607+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:31.550607+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19816448,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10095,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10273","reported_seq":10387,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:29.816224+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:29.816224+0000","last_peered":"2024-09-11T05:04:29.816224+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:29.816224+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20190368,"num_objects":2465,"num_object_clones":0,"num_object_copies":9860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10271,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":207}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9780","reported_seq":9885,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.217462+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:27.217462+0000","last_peered":"2024-09-11T05:04:27.217462+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:27.217462+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9780,"log_dups_size":0,"ondisk_log_size":9780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19495528,"num_objects":2381,"num_object_clones":0,"num_object_copies":9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9779,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":170}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10038","reported_seq":10146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:29.599074+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:29.599074+0000","last_peered":"2024-09-11T05:04:29.599074+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:29.599074+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19477664,"num_objects":2378,"num_object_clones":0,"num_object_copies":9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10036,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":117}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10190","reported_seq":10313,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.624974+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:31.624974+0000","last_peered":"2024-09-11T05:04:31.624974+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:31.624974+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10188,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10251","reported_seq":10356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.632327+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:31.632327+0000","last_peered":"2024-09-11T05:04:31.632327+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:31.632327+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19570688,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10251,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":95}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10286","reported_seq":10380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:29.770274+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:29.770274+0000","last_peered":"2024-09-11T05:04:29.770274+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:29.770274+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20018336,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10284,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":137}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10389","reported_seq":15246,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:26.955177+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:26.955177+0000","last_peered":"2024-09-11T05:04:26.955177+0000","last_clean":"2024-09-11T05:04:26.955177+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:26.955177+0000","last_undegraded":"2024-09-11T05:04:26.955177+0000","last_fullsized":"2024-09-11T05:04:26.955177+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19939328,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10382","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.525574+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:31.525574+0000","last_peered":"2024-09-11T05:04:31.525574+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:31.525574+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19704128,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10379,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":40}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10131","reported_seq":10241,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.375128+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:27.375128+0000","last_peered":"2024-09-11T05:04:27.375128+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:27.375128+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633312,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10129,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":91}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10263","reported_seq":10380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.606092+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:31.606092+0000","last_peered":"2024-09-11T05:04:31.606092+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:31.606092+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19574512,"num_objects":2390,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":92}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9977","reported_seq":14578,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:27.212439+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:27.212439+0000","last_peered":"2024-09-11T05:04:27.212439+0000","last_clean":"2024-09-11T05:04:27.212439+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:27.212439+0000","last_undegraded":"2024-09-11T05:04:27.212439+0000","last_fullsized":"2024-09-11T05:04:27.212439+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":9977,"log_dups_size":0,"ondisk_log_size":9977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19718144,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3787,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10274","reported_seq":12132,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.480056+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:27.480056+0000","last_peered":"2024-09-11T05:04:27.480056+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:27.480056+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060752,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10232","reported_seq":10339,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.270085+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:31.270085+0000","last_peered":"2024-09-11T05:04:31.270085+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:31.270085+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330208,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10230,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10100","reported_seq":10219,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:27.116210+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:27.116210+0000","last_peered":"2024-09-11T05:04:27.116210+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:27.116210+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18912416,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10098,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10327","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:29.713799+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:29.713799+0000","last_peered":"2024-09-11T05:04:29.713799+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:29.713799+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067488,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":314683880,"num_objects":38419,"num_object_clones":0,"num_object_copies":153676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147484,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630853632,"data_stored":630853632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160688,"ondisk_log_size":160688,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747952,"kb_used_data":152608,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93623888,"statfs":{"total":96636764160,"available":95870861312,"internally_reserved":0,"allocated":156270592,"data_stored":156081390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,4,9,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742884,"kb_used_data":156372,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93628956,"statfs":{"total":96636764160,"available":95876050944,"internally_reserved":0,"allocated":160124928,"data_stored":159935726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,0,0,6,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606784,"kb_used_data":155884,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93765056,"statfs":{"total":96636764160,"available":96015417344,"internally_reserved":0,"allocated":159625216,"data_stored":159436014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":461696722},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740644,"kb_used_data":153684,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93631196,"statfs":{"total":96636764160,"available":95878344704,"internally_reserved":0,"allocated":157372416,"data_stored":157183214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156995584,"data_stored":156995584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159256576,"data_stored":159256576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":158978048,"data_stored":158978048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155623424,"data_stored":155623424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:35.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:35.961 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-11T05:04:35.841123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309826024,"num_objects":37826,"num_object_clones":0,"num_object_copies":151304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148077,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1210,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838460,"kb_used_data":606264,"kb_used_omap":14,"kb_used_meta":2232113,"kb_avail":374648900,"statfs":{"total":386547056640,"available":383640473600,"internally_reserved":0,"allocated":620814336,"data_stored":620057528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2285684278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,2,12,14,24,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17694720,"num_objects":-2160,"num_object_clones":0,"num_object_copies":-8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2160,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006727"},"pg_stats":[{"pgid":"1.f","version":"35'10095","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.550607+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:31.550607+0000","last_peered":"2024-09-11T05:04:31.550607+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:31.550607+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19816448,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10095,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10332","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.778980+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:34.778980+0000","last_peered":"2024-09-11T05:04:34.778980+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:34.778980+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19707040,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10330,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9838","reported_seq":9942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:32.350731+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:32.350731+0000","last_peered":"2024-09-11T05:04:32.350731+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:32.350731+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19020392,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9837,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10090","reported_seq":10198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.692564+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:34.692564+0000","last_peered":"2024-09-11T05:04:34.692564+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:34.692564+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19051680,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10088,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10190","reported_seq":10313,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.624974+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:31.624974+0000","last_peered":"2024-09-11T05:04:31.624974+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:31.624974+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10188,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10251","reported_seq":10356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.632327+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:31.632327+0000","last_peered":"2024-09-11T05:04:31.632327+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:31.632327+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19570688,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10251,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":95}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10337","reported_seq":10431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.842858+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:34.842858+0000","last_peered":"2024-09-11T05:04:34.842858+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:34.842858+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19600544,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10451","reported_seq":15308,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:32.220857+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:32.220857+0000","last_peered":"2024-09-11T05:04:32.220857+0000","last_clean":"2024-09-11T05:04:32.220857+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:32.220857+0000","last_undegraded":"2024-09-11T05:04:32.220857+0000","last_fullsized":"2024-09-11T05:04:32.220857+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19431424,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10382","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.525574+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:31.525574+0000","last_peered":"2024-09-11T05:04:31.525574+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:31.525574+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19704128,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10379,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":40}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10200","reported_seq":10312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:32.468126+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:32.468126+0000","last_peered":"2024-09-11T05:04:32.468126+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:32.468126+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19068064,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":22}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10263","reported_seq":10380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.606092+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:31.606092+0000","last_peered":"2024-09-11T05:04:31.606092+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:31.606092+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19574512,"num_objects":2390,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":92}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10023","reported_seq":14624,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:31.911721+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:31.911721+0000","last_peered":"2024-09-11T05:04:31.911721+0000","last_clean":"2024-09-11T05:04:31.911721+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:31.911721+0000","last_undegraded":"2024-09-11T05:04:31.911721+0000","last_fullsized":"2024-09-11T05:04:31.911721+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19341312,"num_objects":2361,"num_object_clones":0,"num_object_copies":9444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10345","reported_seq":12283,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:32.394167+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:32.394167+0000","last_peered":"2024-09-11T05:04:32.394167+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:32.394167+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19479120,"num_objects":2378,"num_object_clones":0,"num_object_copies":9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10344,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10232","reported_seq":10339,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.270085+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:31.270085+0000","last_peered":"2024-09-11T05:04:31.270085+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:31.270085+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330208,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10230,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10160","reported_seq":10279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:32.376791+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:32.376791+0000","last_peered":"2024-09-11T05:04:32.376791+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:32.376791+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18420896,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10158,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-54}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10392","reported_seq":10512,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.705518+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:34.705518+0000","last_peered":"2024-09-11T05:04:34.705518+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:34.705518+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19535008,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":97}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309826024,"num_objects":37826,"num_object_clones":0,"num_object_copies":151304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148077,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1210,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":618962944,"data_stored":618962944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747952,"kb_used_data":152608,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93623888,"statfs":{"total":96636764160,"available":95870861312,"internally_reserved":0,"allocated":156270592,"data_stored":156081390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,4,9,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742424,"kb_used_data":151752,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93629416,"statfs":{"total":96636764160,"available":95876521984,"internally_reserved":0,"allocated":155394048,"data_stored":155204846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607120,"kb_used_data":152064,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93764720,"statfs":{"total":96636764160,"available":96015073280,"internally_reserved":0,"allocated":155713536,"data_stored":155524334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":465956562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740964,"kb_used_data":149840,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93630876,"statfs":{"total":96636764160,"available":95878017024,"internally_reserved":0,"allocated":153436160,"data_stored":153246958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153022464,"data_stored":153022464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155267072,"data_stored":155267072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155049984,"data_stored":155049984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155623424,"data_stored":155623424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:35.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:36.656 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:36.680 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-11T05:04:35.841123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309826024,"num_objects":37826,"num_object_clones":0,"num_object_copies":151304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148077,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1210,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838460,"kb_used_data":606264,"kb_used_omap":14,"kb_used_meta":2232113,"kb_avail":374648900,"statfs":{"total":386547056640,"available":383640473600,"internally_reserved":0,"allocated":620814336,"data_stored":620057528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2285684278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,2,12,14,24,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17694720,"num_objects":-2160,"num_object_clones":0,"num_object_copies":-8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2160,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006727"},"pg_stats":[{"pgid":"1.f","version":"35'10095","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.550607+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:31.550607+0000","last_peered":"2024-09-11T05:04:31.550607+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:31.550607+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19816448,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10095,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10332","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.778980+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:34.778980+0000","last_peered":"2024-09-11T05:04:34.778980+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:34.778980+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19707040,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10330,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9838","reported_seq":9942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:32.350731+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:32.350731+0000","last_peered":"2024-09-11T05:04:32.350731+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:32.350731+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19020392,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9837,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10090","reported_seq":10198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.692564+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:34.692564+0000","last_peered":"2024-09-11T05:04:34.692564+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:34.692564+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19051680,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10088,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10190","reported_seq":10313,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.624974+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:31.624974+0000","last_peered":"2024-09-11T05:04:31.624974+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:31.624974+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19174560,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10188,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10251","reported_seq":10356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.632327+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:31.632327+0000","last_peered":"2024-09-11T05:04:31.632327+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:31.632327+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19570688,"num_objects":2389,"num_object_clones":0,"num_object_copies":9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10251,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":95}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10337","reported_seq":10431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.842858+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:34.842858+0000","last_peered":"2024-09-11T05:04:34.842858+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:34.842858+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19600544,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10451","reported_seq":15308,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:32.220857+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:32.220857+0000","last_peered":"2024-09-11T05:04:32.220857+0000","last_clean":"2024-09-11T05:04:32.220857+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:32.220857+0000","last_undegraded":"2024-09-11T05:04:32.220857+0000","last_fullsized":"2024-09-11T05:04:32.220857+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19431424,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10382","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.525574+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:31.525574+0000","last_peered":"2024-09-11T05:04:31.525574+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:31.525574+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19704128,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10379,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":40}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10200","reported_seq":10312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:32.468126+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:32.468126+0000","last_peered":"2024-09-11T05:04:32.468126+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:32.468126+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19068064,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":22}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10263","reported_seq":10380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.606092+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:31.606092+0000","last_peered":"2024-09-11T05:04:31.606092+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:31.606092+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19574512,"num_objects":2390,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":92}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10023","reported_seq":14624,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:31.911721+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:31.911721+0000","last_peered":"2024-09-11T05:04:31.911721+0000","last_clean":"2024-09-11T05:04:31.911721+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:31.911721+0000","last_undegraded":"2024-09-11T05:04:31.911721+0000","last_fullsized":"2024-09-11T05:04:31.911721+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19341312,"num_objects":2361,"num_object_clones":0,"num_object_copies":9444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10345","reported_seq":12283,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:32.394167+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:32.394167+0000","last_peered":"2024-09-11T05:04:32.394167+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:32.394167+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19479120,"num_objects":2378,"num_object_clones":0,"num_object_copies":9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10344,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10232","reported_seq":10339,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:31.270085+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:31.270085+0000","last_peered":"2024-09-11T05:04:31.270085+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:31.270085+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330208,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10230,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10160","reported_seq":10279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:32.376791+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:32.376791+0000","last_peered":"2024-09-11T05:04:32.376791+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:32.376791+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18420896,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10158,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-54}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10392","reported_seq":10512,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.705518+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:34.705518+0000","last_peered":"2024-09-11T05:04:34.705518+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:34.705518+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19535008,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":97}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309826024,"num_objects":37826,"num_object_clones":0,"num_object_copies":151304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148077,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1210,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":618962944,"data_stored":618962944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747952,"kb_used_data":152608,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93623888,"statfs":{"total":96636764160,"available":95870861312,"internally_reserved":0,"allocated":156270592,"data_stored":156081390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,4,9,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742424,"kb_used_data":151752,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93629416,"statfs":{"total":96636764160,"available":95876521984,"internally_reserved":0,"allocated":155394048,"data_stored":155204846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607120,"kb_used_data":152064,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93764720,"statfs":{"total":96636764160,"available":96015073280,"internally_reserved":0,"allocated":155713536,"data_stored":155524334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":465956562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740964,"kb_used_data":149840,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93630876,"statfs":{"total":96636764160,"available":95878017024,"internally_reserved":0,"allocated":153436160,"data_stored":153246958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153022464,"data_stored":153022464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155267072,"data_stored":155267072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155049984,"data_stored":155049984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155623424,"data_stored":155623424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:36.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:36.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10095", 'reported_seq': 10189, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:31.550607+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:31.550607+0000', 'last_peered': '2024-09-11T05:04:31.550607+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:31.550607+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19816448, 'num_objects': 2419, 'num_object_clones': 0, 'num_object_copies': 9676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10095, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 128}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:36.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:36.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10332", 'reported_seq': 10447, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:34.778980+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:34.778980+0000', 'last_peered': '2024-09-11T05:04:34.778980+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:34.778980+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19707040, 'num_objects': 2406, 'num_object_clones': 0, 'num_object_copies': 9624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10330, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 148}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:36.682 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:36.682 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9838", 'reported_seq': 9942, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:32.350731+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:32.350731+0000', 'last_peered': '2024-09-11T05:04:32.350731+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:32.350731+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9838, 'log_dups_size': 0, 'ondisk_log_size': 9838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19020392, 'num_objects': 2323, 'num_object_clones': 0, 'num_object_copies': 9292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2323, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9837, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 112}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10090", 'reported_seq': 10198, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:34.692564+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:34.692564+0000', 'last_peered': '2024-09-11T05:04:34.692564+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:34.692564+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19051680, 'num_objects': 2326, 'num_object_clones': 0, 'num_object_copies': 9304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10088, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 65}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10190", 'reported_seq': 10313, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:31.624974+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:31.624974+0000', 'last_peered': '2024-09-11T05:04:31.624974+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:31.624974+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19174560, 'num_objects': 2341, 'num_object_clones': 0, 'num_object_copies': 9364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10188, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 120}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10251", 'reported_seq': 10356, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:31.632327+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:31.632327+0000', 'last_peered': '2024-09-11T05:04:31.632327+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:31.632327+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19570688, 'num_objects': 2389, 'num_object_clones': 0, 'num_object_copies': 9556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10251, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 95}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10337", 'reported_seq': 10431, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:34.842858+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:34.842858+0000', 'last_peered': '2024-09-11T05:04:34.842858+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:34.842858+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19600544, 'num_objects': 2393, 'num_object_clones': 0, 'num_object_copies': 9572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10335, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 86}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:36.683 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10382", 'reported_seq': 10505, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:31.525574+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:31.525574+0000', 'last_peered': '2024-09-11T05:04:31.525574+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:31.525574+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19704128, 'num_objects': 2406, 'num_object_clones': 0, 'num_object_copies': 9624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10379, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 40}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10200", 'reported_seq': 10312, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:32.468126+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:32.468126+0000', 'last_peered': '2024-09-11T05:04:32.468126+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:32.468126+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19068064, 'num_objects': 2328, 'num_object_clones': 0, 'num_object_copies': 9312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10198, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 22}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10263", 'reported_seq': 10380, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:31.606092+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:31.606092+0000', 'last_peered': '2024-09-11T05:04:31.606092+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:31.606092+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19574512, 'num_objects': 2390, 'num_object_clones': 0, 'num_object_copies': 9560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10260, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 92}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10345", 'reported_seq': 12283, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:32.394167+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:32.394167+0000', 'last_peered': '2024-09-11T05:04:32.394167+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:32.394167+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19479120, 'num_objects': 2378, 'num_object_clones': 0, 'num_object_copies': 9512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1407, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10344, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 927, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 971}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10232", 'reported_seq': 10339, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:31.270085+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:31.270085+0000', 'last_peered': '2024-09-11T05:04:31.270085+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:31.270085+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19330208, 'num_objects': 2360, 'num_object_clones': 0, 'num_object_copies': 9440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10230, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 96}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:36.684 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10160", 'reported_seq': 10279, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:32.376791+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:32.376791+0000', 'last_peered': '2024-09-11T05:04:32.376791+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:32.376791+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18420896, 'num_objects': 2249, 'num_object_clones': 0, 'num_object_copies': 8996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10158, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -54}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:36.685 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:36.685 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10392", 'reported_seq': 10512, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:34.705518+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:34.705518+0000', 'last_peered': '2024-09-11T05:04:34.705518+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:34.705518+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19535008, 'num_objects': 2385, 'num_object_clones': 0, 'num_object_copies': 9540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10390, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 97}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:36.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:37.481 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:37.481 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31098/151304 objects degraded (20.553%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":604,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37826,"data_bytes":309826024,"bytes_used":2906583040,"bytes_avail":383640473600,"bytes_total":386547056640,"degraded_objects":31098,"degraded_total":151304,"degraded_ratio":0.2055332311108761,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":179},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:37.510 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:37.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:38.165 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:38.165 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:38.198 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-11T05:04:37.841983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":303477224,"num_objects":37051,"num_object_clones":0,"num_object_copies":148204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148852,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857084,"kb_used_data":594324,"kb_used_omap":14,"kb_used_meta":2262641,"kb_avail":374630276,"statfs":{"total":386547056640,"available":383621402624,"internally_reserved":0,"allocated":608587776,"data_stored":607830968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2316944950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,2,5,21,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22118400,"num_objects":-2700,"num_object_clones":0,"num_object_copies":-10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006755"},"pg_stats":[{"pgid":"1.f","version":"35'10167","reported_seq":10262,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.647052+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:36.647052+0000","last_peered":"2024-09-11T05:04:36.647052+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:36.647052+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19226624,"num_objects":2347,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10167,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":56}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10332","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.778980+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:34.778980+0000","last_peered":"2024-09-11T05:04:34.778980+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:34.778980+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19707040,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10330,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9909","reported_seq":10014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.401589+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:37.401589+0000","last_peered":"2024-09-11T05:04:37.401589+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:37.401589+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438760,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9908,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":41}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10090","reported_seq":10198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.692564+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:34.692564+0000","last_peered":"2024-09-11T05:04:34.692564+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:34.692564+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19051680,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10088,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10245","reported_seq":10366,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.717892+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:36.717892+0000","last_peered":"2024-09-11T05:04:36.717892+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:36.717892+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18724000,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10243,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":65}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10325","reported_seq":10430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.554627+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:36.554627+0000","last_peered":"2024-09-11T05:04:36.554627+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:36.554627+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18964480,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":21}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10337","reported_seq":10431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.842858+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:34.842858+0000","last_peered":"2024-09-11T05:04:34.842858+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:34.842858+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19600544,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10512","reported_seq":15369,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:37.375481+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:37.375481+0000","last_peered":"2024-09-11T05:04:37.375481+0000","last_clean":"2024-09-11T05:04:37.375481+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:37.375481+0000","last_undegraded":"2024-09-11T05:04:37.375481+0000","last_fullsized":"2024-09-11T05:04:37.375481+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18931712,"num_objects":2311,"num_object_clones":0,"num_object_copies":9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1222,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10441","reported_seq":10564,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.737069+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:36.737069+0000","last_peered":"2024-09-11T05:04:36.737069+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:36.737069+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19220800,"num_objects":2347,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10438,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-19}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10266","reported_seq":10377,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.225681+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:37.225681+0000","last_peered":"2024-09-11T05:04:37.225681+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:37.225681+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527392,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-44}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10330","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.635795+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:36.635795+0000","last_peered":"2024-09-11T05:04:36.635795+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:36.635795+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19025648,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":25}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10067","reported_seq":14668,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:36.633694+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:36.633694+0000","last_peered":"2024-09-11T05:04:36.633694+0000","last_clean":"2024-09-11T05:04:36.633694+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:36.633694+0000","last_undegraded":"2024-09-11T05:04:36.633694+0000","last_fullsized":"2024-09-11T05:04:36.633694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18980864,"num_objects":2317,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3877,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10422","reported_seq":12453,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.525867+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:37.525867+0000","last_peered":"2024-09-11T05:04:37.525867+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:37.525867+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18848336,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10421,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":975,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":942}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10293","reported_seq":10400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.451648+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:36.451648+0000","last_peered":"2024-09-11T05:04:36.451648+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:36.451648+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18830496,"num_objects":2299,"num_object_clones":0,"num_object_copies":9196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":35}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10228","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.208430+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:37.208430+0000","last_peered":"2024-09-11T05:04:37.208430+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:37.208430+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10392","reported_seq":10512,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:34.705518+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:34.705518+0000","last_peered":"2024-09-11T05:04:34.705518+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:34.705518+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19535008,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":97}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":303477224,"num_objects":37051,"num_object_clones":0,"num_object_copies":148204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148852,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":606679040,"data_stored":606679040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748128,"kb_used_data":148620,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93623712,"statfs":{"total":96636764160,"available":95870681088,"internally_reserved":0,"allocated":152186880,"data_stored":151997678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742652,"kb_used_data":147820,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93629188,"statfs":{"total":96636764160,"available":95876288512,"internally_reserved":0,"allocated":151367680,"data_stored":151178478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,6,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625340,"kb_used_data":148044,"kb_used_omap":4,"kb_used_meta":477243,"kb_avail":93746500,"statfs":{"total":96636764160,"available":95996416000,"internally_reserved":0,"allocated":151597056,"data_stored":151407854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":488697554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,11,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740964,"kb_used_data":149840,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93630876,"statfs":{"total":96636764160,"available":95878017024,"internally_reserved":0,"allocated":153436160,"data_stored":153246958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153022464,"data_stored":153022464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151080960,"data_stored":151080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151007232,"data_stored":151007232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151568384,"data_stored":151568384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:41.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:41.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:41.897 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:41.913 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-11T05:04:41.843356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":301478376,"num_objects":36807,"num_object_clones":0,"num_object_copies":147228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36807,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149096,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857456,"kb_used_data":590540,"kb_used_omap":14,"kb_used_meta":2266801,"kb_avail":374629904,"statfs":{"total":386547056640,"available":383621021696,"internally_reserved":0,"allocated":604712960,"data_stored":603956152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2321204790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,4,4,21,19,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18235392,"num_objects":-2226,"num_object_clones":0,"num_object_copies":-8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006758"},"pg_stats":[{"pgid":"1.f","version":"35'10167","reported_seq":10262,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.647052+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:36.647052+0000","last_peered":"2024-09-11T05:04:36.647052+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:36.647052+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19226624,"num_objects":2347,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10167,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":56}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10388","reported_seq":10503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.957253+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:39.957253+0000","last_peered":"2024-09-11T05:04:39.957253+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:39.957253+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248288,"num_objects":2350,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10386,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":92}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9909","reported_seq":10014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.401589+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:37.401589+0000","last_peered":"2024-09-11T05:04:37.401589+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:37.401589+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438760,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9908,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":41}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10164","reported_seq":10271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.892268+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:39.892268+0000","last_peered":"2024-09-11T05:04:39.892268+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:39.892268+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18445472,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10245","reported_seq":10366,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.717892+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:36.717892+0000","last_peered":"2024-09-11T05:04:36.717892+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:36.717892+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18724000,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10243,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":65}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10325","reported_seq":10430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.554627+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:36.554627+0000","last_peered":"2024-09-11T05:04:36.554627+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:36.554627+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18964480,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":21}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10393","reported_seq":10487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.844785+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:39.844785+0000","last_peered":"2024-09-11T05:04:39.844785+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:39.844785+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19141792,"num_objects":2337,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10391,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":30}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10512","reported_seq":15369,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:37.375481+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:37.375481+0000","last_peered":"2024-09-11T05:04:37.375481+0000","last_clean":"2024-09-11T05:04:37.375481+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:37.375481+0000","last_undegraded":"2024-09-11T05:04:37.375481+0000","last_fullsized":"2024-09-11T05:04:37.375481+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18931712,"num_objects":2311,"num_object_clones":0,"num_object_copies":9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1222,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10441","reported_seq":10564,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.737069+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:36.737069+0000","last_peered":"2024-09-11T05:04:36.737069+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:36.737069+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19220800,"num_objects":2347,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10438,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-19}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10266","reported_seq":10377,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.225681+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:37.225681+0000","last_peered":"2024-09-11T05:04:37.225681+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:37.225681+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527392,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-44}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10330","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.635795+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:36.635795+0000","last_peered":"2024-09-11T05:04:36.635795+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:36.635795+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19025648,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":25}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10067","reported_seq":14668,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:36.633694+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:36.633694+0000","last_peered":"2024-09-11T05:04:36.633694+0000","last_clean":"2024-09-11T05:04:36.633694+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:36.633694+0000","last_undegraded":"2024-09-11T05:04:36.633694+0000","last_fullsized":"2024-09-11T05:04:36.633694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18980864,"num_objects":2317,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3877,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10422","reported_seq":12453,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.525867+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:37.525867+0000","last_peered":"2024-09-11T05:04:37.525867+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:37.525867+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18848336,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10421,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":975,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":942}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10293","reported_seq":10400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.451648+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:36.451648+0000","last_peered":"2024-09-11T05:04:36.451648+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:36.451648+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18830496,"num_objects":2299,"num_object_clones":0,"num_object_copies":9196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":35}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10228","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.208430+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:37.208430+0000","last_peered":"2024-09-11T05:04:37.208430+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:37.208430+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10450","reported_seq":10570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.760631+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:39.760631+0000","last_peered":"2024-09-11T05:04:39.760631+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:39.760631+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19059872,"num_objects":2327,"num_object_clones":0,"num_object_copies":9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":39}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":301478376,"num_objects":36807,"num_object_clones":0,"num_object_copies":147228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36807,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149096,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602652672,"data_stored":602652672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748128,"kb_used_data":148620,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93623712,"statfs":{"total":96636764160,"available":95870681088,"internally_reserved":0,"allocated":152186880,"data_stored":151997678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742652,"kb_used_data":147820,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93629188,"statfs":{"total":96636764160,"available":95876288512,"internally_reserved":0,"allocated":151367680,"data_stored":151178478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,6,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625340,"kb_used_data":148044,"kb_used_omap":4,"kb_used_meta":477243,"kb_avail":93746500,"statfs":{"total":96636764160,"available":95996416000,"internally_reserved":0,"allocated":151597056,"data_stored":151407854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":488697554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,11,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741336,"kb_used_data":146056,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93630504,"statfs":{"total":96636764160,"available":95877636096,"internally_reserved":0,"allocated":149561344,"data_stored":149372142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148996096,"data_stored":148996096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151080960,"data_stored":151080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151007232,"data_stored":151007232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151568384,"data_stored":151568384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:41.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:42.507 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:42.508 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:42.536 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-11T05:04:41.843356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":301478376,"num_objects":36807,"num_object_clones":0,"num_object_copies":147228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36807,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149096,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2857456,"kb_used_data":590540,"kb_used_omap":14,"kb_used_meta":2266801,"kb_avail":374629904,"statfs":{"total":386547056640,"available":383621021696,"internally_reserved":0,"allocated":604712960,"data_stored":603956152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2321204790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,4,4,21,19,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18235392,"num_objects":-2226,"num_object_clones":0,"num_object_copies":-8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006758"},"pg_stats":[{"pgid":"1.f","version":"35'10167","reported_seq":10262,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.647052+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:36.647052+0000","last_peered":"2024-09-11T05:04:36.647052+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:36.647052+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19226624,"num_objects":2347,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10167,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":56}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10388","reported_seq":10503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.957253+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:39.957253+0000","last_peered":"2024-09-11T05:04:39.957253+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:39.957253+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248288,"num_objects":2350,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10386,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":92}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9909","reported_seq":10014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.401589+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:37.401589+0000","last_peered":"2024-09-11T05:04:37.401589+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:37.401589+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18438760,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9908,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":41}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10164","reported_seq":10271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.892268+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:39.892268+0000","last_peered":"2024-09-11T05:04:39.892268+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:39.892268+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18445472,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10245","reported_seq":10366,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.717892+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:36.717892+0000","last_peered":"2024-09-11T05:04:36.717892+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:36.717892+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18724000,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10243,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":65}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10325","reported_seq":10430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.554627+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:36.554627+0000","last_peered":"2024-09-11T05:04:36.554627+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:36.554627+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18964480,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":21}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10393","reported_seq":10487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.844785+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:39.844785+0000","last_peered":"2024-09-11T05:04:39.844785+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:39.844785+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19141792,"num_objects":2337,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10391,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":30}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10512","reported_seq":15369,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:37.375481+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:37.375481+0000","last_peered":"2024-09-11T05:04:37.375481+0000","last_clean":"2024-09-11T05:04:37.375481+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:37.375481+0000","last_undegraded":"2024-09-11T05:04:37.375481+0000","last_fullsized":"2024-09-11T05:04:37.375481+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18931712,"num_objects":2311,"num_object_clones":0,"num_object_copies":9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1222,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10441","reported_seq":10564,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.737069+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:36.737069+0000","last_peered":"2024-09-11T05:04:36.737069+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:36.737069+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19220800,"num_objects":2347,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10438,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-19}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10266","reported_seq":10377,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.225681+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:37.225681+0000","last_peered":"2024-09-11T05:04:37.225681+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:37.225681+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527392,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-44}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10330","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.635795+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:36.635795+0000","last_peered":"2024-09-11T05:04:36.635795+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:36.635795+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19025648,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":25}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10067","reported_seq":14668,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:36.633694+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:36.633694+0000","last_peered":"2024-09-11T05:04:36.633694+0000","last_clean":"2024-09-11T05:04:36.633694+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:36.633694+0000","last_undegraded":"2024-09-11T05:04:36.633694+0000","last_fullsized":"2024-09-11T05:04:36.633694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18980864,"num_objects":2317,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3877,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10422","reported_seq":12453,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.525867+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:37.525867+0000","last_peered":"2024-09-11T05:04:37.525867+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:37.525867+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18848336,"num_objects":2301,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10421,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":975,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":942}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10293","reported_seq":10400,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:36.451648+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:36.451648+0000","last_peered":"2024-09-11T05:04:36.451648+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:36.451648+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18830496,"num_objects":2299,"num_object_clones":0,"num_object_copies":9196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":35}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10228","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:37.208430+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:37.208430+0000","last_peered":"2024-09-11T05:04:37.208430+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:37.208430+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10450","reported_seq":10570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.760631+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:39.760631+0000","last_peered":"2024-09-11T05:04:39.760631+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:39.760631+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19059872,"num_objects":2327,"num_object_clones":0,"num_object_copies":9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":39}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":301478376,"num_objects":36807,"num_object_clones":0,"num_object_copies":147228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36807,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149096,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602652672,"data_stored":602652672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748128,"kb_used_data":148620,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93623712,"statfs":{"total":96636764160,"available":95870681088,"internally_reserved":0,"allocated":152186880,"data_stored":151997678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742652,"kb_used_data":147820,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93629188,"statfs":{"total":96636764160,"available":95876288512,"internally_reserved":0,"allocated":151367680,"data_stored":151178478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,6,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625340,"kb_used_data":148044,"kb_used_omap":4,"kb_used_meta":477243,"kb_avail":93746500,"statfs":{"total":96636764160,"available":95996416000,"internally_reserved":0,"allocated":151597056,"data_stored":151407854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":488697554},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,11,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741336,"kb_used_data":146056,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93630504,"statfs":{"total":96636764160,"available":95877636096,"internally_reserved":0,"allocated":149561344,"data_stored":149372142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148996096,"data_stored":148996096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151080960,"data_stored":151080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151007232,"data_stored":151007232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151568384,"data_stored":151568384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:42.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:42.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10167", 'reported_seq': 10262, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:36.647052+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:36.647052+0000', 'last_peered': '2024-09-11T05:04:36.647052+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:36.647052+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19226624, 'num_objects': 2347, 'num_object_clones': 0, 'num_object_copies': 9388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2347, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10167, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 56}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10388", 'reported_seq': 10503, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:39.957253+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:39.957253+0000', 'last_peered': '2024-09-11T05:04:39.957253+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:39.957253+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19248288, 'num_objects': 2350, 'num_object_clones': 0, 'num_object_copies': 9400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10386, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 92}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9909", 'reported_seq': 10014, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:37.401589+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:37.401589+0000', 'last_peered': '2024-09-11T05:04:37.401589+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:37.401589+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 9909, 'log_dups_size': 0, 'ondisk_log_size': 9909, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18438760, 'num_objects': 2252, 'num_object_clones': 0, 'num_object_copies': 9008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2252, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9908, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 41}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10164", 'reported_seq': 10271, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:39.892268+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:39.892268+0000', 'last_peered': '2024-09-11T05:04:39.892268+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:39.892268+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18445472, 'num_objects': 2252, 'num_object_clones': 0, 'num_object_copies': 9008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10162, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -9}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10245", 'reported_seq': 10366, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:36.717892+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:36.717892+0000', 'last_peered': '2024-09-11T05:04:36.717892+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:36.717892+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18724000, 'num_objects': 2286, 'num_object_clones': 0, 'num_object_copies': 9144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10243, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 65}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:42.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10325", 'reported_seq': 10430, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:36.554627+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:36.554627+0000', 'last_peered': '2024-09-11T05:04:36.554627+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:36.554627+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18964480, 'num_objects': 2315, 'num_object_clones': 0, 'num_object_copies': 9260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10325, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 21}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10393", 'reported_seq': 10487, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:39.844785+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:39.844785+0000', 'last_peered': '2024-09-11T05:04:39.844785+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:39.844785+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19141792, 'num_objects': 2337, 'num_object_clones': 0, 'num_object_copies': 9348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10391, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 30}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10441", 'reported_seq': 10564, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:36.737069+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:36.737069+0000', 'last_peered': '2024-09-11T05:04:36.737069+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:36.737069+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19220800, 'num_objects': 2347, 'num_object_clones': 0, 'num_object_copies': 9388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2347, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10438, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -19}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10266", 'reported_seq': 10377, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:37.225681+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:37.225681+0000', 'last_peered': '2024-09-11T05:04:37.225681+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:37.225681+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18527392, 'num_objects': 2262, 'num_object_clones': 0, 'num_object_copies': 9048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10264, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -44}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10330", 'reported_seq': 10447, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:36.635795+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:36.635795+0000', 'last_peered': '2024-09-11T05:04:36.635795+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:36.635795+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19025648, 'num_objects': 2323, 'num_object_clones': 0, 'num_object_copies': 9292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10327, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 25}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:42.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10422", 'reported_seq': 12453, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:37.525867+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:37.525867+0000', 'last_peered': '2024-09-11T05:04:37.525867+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:37.525867+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18848336, 'num_objects': 2301, 'num_object_clones': 0, 'num_object_copies': 9204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10421, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 975, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 942}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:42.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:42.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10293", 'reported_seq': 10400, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:36.451648+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:36.451648+0000', 'last_peered': '2024-09-11T05:04:36.451648+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:36.451648+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18830496, 'num_objects': 2299, 'num_object_clones': 0, 'num_object_copies': 9196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10291, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 35}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:42.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:42.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10228", 'reported_seq': 10347, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:37.208430+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:37.208430+0000', 'last_peered': '2024-09-11T05:04:37.208430+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:37.208430+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17863840, 'num_objects': 2181, 'num_object_clones': 0, 'num_object_copies': 8724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10226, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:42.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:42.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10450", 'reported_seq': 10570, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:39.760631+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:39.760631+0000', 'last_peered': '2024-09-11T05:04:39.760631+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:39.760631+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19059872, 'num_objects': 2327, 'num_object_clones': 0, 'num_object_copies': 9308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10448, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 39}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:42.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:43.286 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:43.287 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31050/147228 objects degraded (21.090%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":610,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36807,"data_bytes":301478376,"bytes_used":2926034944,"bytes_avail":383621021696,"bytes_total":386547056640,"degraded_objects":31050,"degraded_total":147228,"degraded_ratio":0.21089738364984922,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":185},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:43.308 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:43.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:43.988 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:44.012 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-11T05:04:43.844791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":295285224,"num_objects":36051,"num_object_clones":0,"num_object_copies":144204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149852,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160756,"ondisk_log_size":160756,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2837368,"kb_used_data":578768,"kb_used_omap":14,"kb_used_meta":2258481,"kb_avail":374649992,"statfs":{"total":386547056640,"available":383641591808,"internally_reserved":0,"allocated":592658432,"data_stored":591901624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2312685110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,11,14,15,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19398656,"num_objects":-2368,"num_object_clones":0,"num_object_copies":-9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2368,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006610"},"pg_stats":[{"pgid":"1.f","version":"35'10237","reported_seq":10332,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.706836+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:41.706836+0000","last_peered":"2024-09-11T05:04:41.706836+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:41.706836+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18653184,"num_objects":2277,"num_object_clones":0,"num_object_copies":9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10237,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10388","reported_seq":10503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.957253+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:39.957253+0000","last_peered":"2024-09-11T05:04:39.957253+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:39.957253+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248288,"num_objects":2350,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10386,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":92}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9974","reported_seq":10078,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:42.480754+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:42.480754+0000","last_peered":"2024-09-11T05:04:42.480754+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:42.480754+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9974,"log_dups_size":0,"ondisk_log_size":9974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17906280,"num_objects":2187,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9973,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10164","reported_seq":10271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.892268+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:39.892268+0000","last_peered":"2024-09-11T05:04:39.892268+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:39.892268+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18445472,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-9}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10309","reported_seq":10432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.804906+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:41.804906+0000","last_peered":"2024-09-11T05:04:41.804906+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:41.804906+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18199712,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10403","reported_seq":10509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.645115+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:41.645115+0000","last_peered":"2024-09-11T05:04:41.645115+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:41.645115+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18325504,"num_objects":2237,"num_object_clones":0,"num_object_copies":8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10393","reported_seq":10487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.844785+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:39.844785+0000","last_peered":"2024-09-11T05:04:39.844785+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:39.844785+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19141792,"num_objects":2337,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10391,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":30}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10580","reported_seq":15437,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:42.572387+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:42.572387+0000","last_peered":"2024-09-11T05:04:42.572387+0000","last_clean":"2024-09-11T05:04:42.572387+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:42.572387+0000","last_undegraded":"2024-09-11T05:04:42.572387+0000","last_fullsized":"2024-09-11T05:04:42.572387+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18374656,"num_objects":2243,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10502","reported_seq":10625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.547446+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:41.547446+0000","last_peered":"2024-09-11T05:04:41.547446+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:41.547446+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18721088,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10499,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-80}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10325","reported_seq":10437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:42.416878+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:42.416878+0000","last_peered":"2024-09-11T05:04:42.416878+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:42.416878+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18044064,"num_objects":2203,"num_object_clones":0,"num_object_copies":8812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10323,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-103}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10392","reported_seq":10509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.431827+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:41.431827+0000","last_peered":"2024-09-11T05:04:41.431827+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:41.431827+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18517744,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-37}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10117","reported_seq":14718,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:42.409217+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:42.409217+0000","last_peered":"2024-09-11T05:04:42.409217+0000","last_clean":"2024-09-11T05:04:42.409217+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:42.409217+0000","last_undegraded":"2024-09-11T05:04:42.409217+0000","last_fullsized":"2024-09-11T05:04:42.409217+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18571264,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3927,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10486","reported_seq":12626,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:42.525836+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:42.525836+0000","last_peered":"2024-09-11T05:04:42.525836+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:42.525836+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18324048,"num_objects":2237,"num_object_clones":0,"num_object_copies":8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10485,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1028,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10347","reported_seq":10454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.409657+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:41.409657+0000","last_peered":"2024-09-11T05:04:41.409657+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:41.409657+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18388128,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10345,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-19}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10289","reported_seq":10406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:42.523187+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:42.523187+0000","last_peered":"2024-09-11T05:04:42.523187+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:42.523187+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17364128,"num_objects":2120,"num_object_clones":0,"num_object_copies":8480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10287,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10450","reported_seq":10570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:39.760631+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:39.760631+0000","last_peered":"2024-09-11T05:04:39.760631+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:39.760631+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19059872,"num_objects":2327,"num_object_clones":0,"num_object_copies":9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":39}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":295285224,"num_objects":36051,"num_object_clones":0,"num_object_copies":144204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149852,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":590405632,"data_stored":590405632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160756,"ondisk_log_size":160756,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744180,"kb_used_data":144644,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93627660,"statfs":{"total":96636764160,"available":95874723840,"internally_reserved":0,"allocated":148115456,"data_stored":147926254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,6,1,2,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742940,"kb_used_data":143940,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93628900,"statfs":{"total":96636764160,"available":95875993600,"internally_reserved":0,"allocated":147394560,"data_stored":147205358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608912,"kb_used_data":144128,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93762928,"statfs":{"total":96636764160,"available":96013238272,"internally_reserved":0,"allocated":147587072,"data_stored":147397870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475918034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741336,"kb_used_data":146056,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93630504,"statfs":{"total":96636764160,"available":95877636096,"internally_reserved":0,"allocated":149561344,"data_stored":149372142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148996096,"data_stored":148996096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146997248,"data_stored":146997248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146944000,"data_stored":146944000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":147468288,"data_stored":147468288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:47.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:47.696 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:47.727 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-11T05:04:45.845683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293212648,"num_objects":35798,"num_object_clones":0,"num_object_copies":143192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35798,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150105,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160609,"ondisk_log_size":160609,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2837696,"kb_used_data":574936,"kb_used_omap":14,"kb_used_meta":2262641,"kb_avail":374649664,"statfs":{"total":386547056640,"available":383641255936,"internally_reserved":0,"allocated":588734464,"data_stored":587977656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2316944950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,11,14,15,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18472960,"num_objects":-2255,"num_object_clones":0,"num_object_copies":-9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005282"},"pg_stats":[{"pgid":"1.f","version":"35'10237","reported_seq":10332,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.706836+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:41.706836+0000","last_peered":"2024-09-11T05:04:41.706836+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:41.706836+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18653184,"num_objects":2277,"num_object_clones":0,"num_object_copies":9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10237,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10445","reported_seq":10560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.984323+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:44.984323+0000","last_peered":"2024-09-11T05:04:44.984323+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:44.984323+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18781344,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10443,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":35}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9974","reported_seq":10078,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:42.480754+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:42.480754+0000","last_peered":"2024-09-11T05:04:42.480754+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:42.480754+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":9974,"log_dups_size":0,"ondisk_log_size":9974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17906280,"num_objects":2187,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9973,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10233","reported_seq":10338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.855410+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:44.855410+0000","last_peered":"2024-09-11T05:04:44.855410+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:44.855410+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17880224,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10309","reported_seq":10432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.804906+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:41.804906+0000","last_peered":"2024-09-11T05:04:41.804906+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:41.804906+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18199712,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10403","reported_seq":10509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.645115+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:41.645115+0000","last_peered":"2024-09-11T05:04:41.645115+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:41.645115+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18325504,"num_objects":2237,"num_object_clones":0,"num_object_copies":8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10450","reported_seq":10544,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.727049+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:44.727049+0000","last_peered":"2024-09-11T05:04:44.727049+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:44.727049+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18674848,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-27}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10580","reported_seq":15437,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:42.572387+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:42.572387+0000","last_peered":"2024-09-11T05:04:42.572387+0000","last_clean":"2024-09-11T05:04:42.572387+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:42.572387+0000","last_undegraded":"2024-09-11T05:04:42.572387+0000","last_fullsized":"2024-09-11T05:04:42.572387+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18374656,"num_objects":2243,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10502","reported_seq":10625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.547446+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:41.547446+0000","last_peered":"2024-09-11T05:04:41.547446+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:41.547446+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18721088,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10499,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-80}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10325","reported_seq":10437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:42.416878+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:42.416878+0000","last_peered":"2024-09-11T05:04:42.416878+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:42.416878+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18044064,"num_objects":2203,"num_object_clones":0,"num_object_copies":8812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10323,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-103}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10392","reported_seq":10509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.431827+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:41.431827+0000","last_peered":"2024-09-11T05:04:41.431827+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:41.431827+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18517744,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-37}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10117","reported_seq":14718,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:42.409217+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:42.409217+0000","last_peered":"2024-09-11T05:04:42.409217+0000","last_clean":"2024-09-11T05:04:42.409217+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:42.409217+0000","last_undegraded":"2024-09-11T05:04:42.409217+0000","last_fullsized":"2024-09-11T05:04:42.409217+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18571264,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3927,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10486","reported_seq":12626,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:42.525836+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:42.525836+0000","last_peered":"2024-09-11T05:04:42.525836+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:42.525836+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18324048,"num_objects":2237,"num_object_clones":0,"num_object_copies":8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10485,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1028,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10347","reported_seq":10454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:41.409657+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:41.409657+0000","last_peered":"2024-09-11T05:04:41.409657+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:41.409657+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18388128,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10345,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-19}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10289","reported_seq":10406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:42.523187+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:42.523187+0000","last_peered":"2024-09-11T05:04:42.523187+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:42.523187+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17364128,"num_objects":2120,"num_object_clones":0,"num_object_copies":8480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10287,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10520","reported_seq":10640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.883622+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:44.883622+0000","last_peered":"2024-09-11T05:04:44.883622+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:44.883622+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486432,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10518,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-31}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":293212648,"num_objects":35798,"num_object_clones":0,"num_object_copies":143192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35798,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150105,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":586457088,"data_stored":586457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160609,"ondisk_log_size":160609,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744180,"kb_used_data":144644,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93627660,"statfs":{"total":96636764160,"available":95874723840,"internally_reserved":0,"allocated":148115456,"data_stored":147926254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,6,1,2,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742940,"kb_used_data":143940,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93628900,"statfs":{"total":96636764160,"available":95875993600,"internally_reserved":0,"allocated":147394560,"data_stored":147205358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608912,"kb_used_data":144128,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93762928,"statfs":{"total":96636764160,"available":96013238272,"internally_reserved":0,"allocated":147587072,"data_stored":147397870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475918034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741664,"kb_used_data":142224,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93630176,"statfs":{"total":96636764160,"available":95877300224,"internally_reserved":0,"allocated":145637376,"data_stored":145448174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145047552,"data_stored":145047552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146997248,"data_stored":146997248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146944000,"data_stored":146944000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":147468288,"data_stored":147468288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:47.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:48.387 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:48.388 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:48.407 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-11T05:04:47.846740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":288346600,"num_objects":35204,"num_object_clones":0,"num_object_copies":140816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35204,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150699,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2858984,"kb_used_data":567080,"kb_used_omap":14,"kb_used_meta":2291825,"kb_avail":374628376,"statfs":{"total":386547056640,"available":383619457024,"internally_reserved":0,"allocated":580689920,"data_stored":579933112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2346829366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,4,2,3,23,23,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21479424,"num_objects":-2622,"num_object_clones":0,"num_object_copies":-10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2622,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005617"},"pg_stats":[{"pgid":"1.f","version":"35'10300","reported_seq":10392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.935904+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:46.935904+0000","last_peered":"2024-09-11T05:04:46.935904+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:46.935904+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18137088,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10300,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-77}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10445","reported_seq":10560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.984323+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:44.984323+0000","last_peered":"2024-09-11T05:04:44.984323+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:44.984323+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18781344,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10443,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":35}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10020","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.344003+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:47.344003+0000","last_peered":"2024-09-11T05:04:47.344003+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:47.344003+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17529448,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10019,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-70}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10233","reported_seq":10338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.855410+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:44.855410+0000","last_peered":"2024-09-11T05:04:44.855410+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:44.855410+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17880224,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10354","reported_seq":10477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.537120+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:46.537120+0000","last_peered":"2024-09-11T05:04:46.537120+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:46.537120+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17831072,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10466","reported_seq":10571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.622872+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:46.622872+0000","last_peered":"2024-09-11T05:04:46.622872+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:46.622872+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17809408,"num_objects":2174,"num_object_clones":0,"num_object_copies":8696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10450","reported_seq":10544,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.727049+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:44.727049+0000","last_peered":"2024-09-11T05:04:44.727049+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:44.727049+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18674848,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-27}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10580","reported_seq":15437,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:42.572387+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:42.572387+0000","last_peered":"2024-09-11T05:04:42.572387+0000","last_clean":"2024-09-11T05:04:42.572387+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:42.572387+0000","last_undegraded":"2024-09-11T05:04:42.572387+0000","last_fullsized":"2024-09-11T05:04:42.572387+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18374656,"num_objects":2243,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10565","reported_seq":10687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.864511+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:46.864511+0000","last_peered":"2024-09-11T05:04:46.864511+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:46.864511+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18204992,"num_objects":2223,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10562,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10386","reported_seq":10498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.498960+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:47.498960+0000","last_peered":"2024-09-11T05:04:47.498960+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:47.498960+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17544352,"num_objects":2142,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10384,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10457","reported_seq":10574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.793956+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:46.793956+0000","last_peered":"2024-09-11T05:04:46.793956+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:46.793956+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17985264,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10117","reported_seq":14718,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:42.409217+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:42.409217+0000","last_peered":"2024-09-11T05:04:42.409217+0000","last_clean":"2024-09-11T05:04:42.409217+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:42.409217+0000","last_undegraded":"2024-09-11T05:04:42.409217+0000","last_fullsized":"2024-09-11T05:04:42.409217+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18571264,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3927,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10546","reported_seq":12786,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.598146+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:47.598146+0000","last_peered":"2024-09-11T05:04:47.598146+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:47.598146+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17832528,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10545,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":921}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10411","reported_seq":10518,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.785669+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:46.785669+0000","last_peered":"2024-09-11T05:04:46.785669+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:46.785669+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10409,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-83}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10353","reported_seq":10472,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.433308+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:47.433308+0000","last_peered":"2024-09-11T05:04:47.433308+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:47.433308+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-247}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10520","reported_seq":10640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.883622+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:44.883622+0000","last_peered":"2024-09-11T05:04:44.883622+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:44.883622+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486432,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10518,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-31}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":288346600,"num_objects":35204,"num_object_clones":0,"num_object_copies":140816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35204,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150699,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":578269184,"data_stored":578269184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763844,"kb_used_data":140724,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93607996,"statfs":{"total":96636764160,"available":95854587904,"internally_reserved":0,"allocated":144101376,"data_stored":143912174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,0,12,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744564,"kb_used_data":140004,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93627276,"statfs":{"total":96636764160,"available":95874330624,"internally_reserved":0,"allocated":143364096,"data_stored":143174894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,0,1,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608912,"kb_used_data":144128,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93762928,"statfs":{"total":96636764160,"available":96013238272,"internally_reserved":0,"allocated":147587072,"data_stored":147397870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475918034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741664,"kb_used_data":142224,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93630176,"statfs":{"total":96636764160,"available":95877300224,"internally_reserved":0,"allocated":145637376,"data_stored":145448174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145047552,"data_stored":145047552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146997248,"data_stored":146997248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":142876672,"data_stored":142876672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":143347712,"data_stored":143347712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:48.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10300", 'reported_seq': 10392, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:46.935904+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:46.935904+0000', 'last_peered': '2024-09-11T05:04:46.935904+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:46.935904+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18137088, 'num_objects': 2214, 'num_object_clones': 0, 'num_object_copies': 8856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10300, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -77}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10445", 'reported_seq': 10560, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:44.984323+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:44.984323+0000', 'last_peered': '2024-09-11T05:04:44.984323+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:44.984323+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18781344, 'num_objects': 2293, 'num_object_clones': 0, 'num_object_copies': 9172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10443, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 35}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10020", 'reported_seq': 10125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:47.344003+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:47.344003+0000', 'last_peered': '2024-09-11T05:04:47.344003+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:47.344003+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17529448, 'num_objects': 2141, 'num_object_clones': 0, 'num_object_copies': 8564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2141, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10019, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -70}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10233", 'reported_seq': 10338, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:44.855410+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:44.855410+0000', 'last_peered': '2024-09-11T05:04:44.855410+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:44.855410+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17880224, 'num_objects': 2183, 'num_object_clones': 0, 'num_object_copies': 8732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10231, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -78}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:48.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10354", 'reported_seq': 10477, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:46.537120+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:46.537120+0000', 'last_peered': '2024-09-11T05:04:46.537120+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:46.537120+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17831072, 'num_objects': 2177, 'num_object_clones': 0, 'num_object_copies': 8708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10352, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -44}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:48.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:48.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10466", 'reported_seq': 10571, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:46.622872+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:46.622872+0000', 'last_peered': '2024-09-11T05:04:46.622872+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:46.622872+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17809408, 'num_objects': 2174, 'num_object_clones': 0, 'num_object_copies': 8696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10466, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -120}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:48.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:48.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10450", 'reported_seq': 10544, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:44.727049+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:44.727049+0000', 'last_peered': '2024-09-11T05:04:44.727049+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:44.727049+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18674848, 'num_objects': 2280, 'num_object_clones': 0, 'num_object_copies': 9120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10448, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -27}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:48.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:48.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10565", 'reported_seq': 10687, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:46.864511+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:46.864511+0000', 'last_peered': '2024-09-11T05:04:46.864511+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:46.864511+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18204992, 'num_objects': 2223, 'num_object_clones': 0, 'num_object_copies': 8892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10562, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:48.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10386", 'reported_seq': 10498, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:47.498960+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:47.498960+0000', 'last_peered': '2024-09-11T05:04:47.498960+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:47.498960+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17544352, 'num_objects': 2142, 'num_object_clones': 0, 'num_object_copies': 8568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10384, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -164}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10457", 'reported_seq': 10574, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:46.793956+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:46.793956+0000', 'last_peered': '2024-09-11T05:04:46.793956+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:46.793956+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17985264, 'num_objects': 2196, 'num_object_clones': 0, 'num_object_copies': 8784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10454, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -102}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10546", 'reported_seq': 12786, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:47.598146+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:47.598146+0000', 'last_peered': '2024-09-11T05:04:47.598146+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:47.598146+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17832528, 'num_objects': 2177, 'num_object_clones': 0, 'num_object_copies': 8708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1256, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10545, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1078, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 921}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10411", 'reported_seq': 10518, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:46.785669+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:46.785669+0000', 'last_peered': '2024-09-11T05:04:46.785669+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:46.785669+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17863840, 'num_objects': 2181, 'num_object_clones': 0, 'num_object_copies': 8724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10409, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -83}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:48.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10353", 'reported_seq': 10472, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:47.433308+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:47.433308+0000', 'last_peered': '2024-09-11T05:04:47.433308+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:47.433308+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16839840, 'num_objects': 2056, 'num_object_clones': 0, 'num_object_copies': 8224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10351, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -247}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:48.412 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:48.412 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10520", 'reported_seq': 10640, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:44.883622+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:44.883622+0000', 'last_peered': '2024-09-11T05:04:44.883622+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:44.883622+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18486432, 'num_objects': 2257, 'num_object_clones': 0, 'num_object_copies': 9028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10518, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -31}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:48.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:49.213 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30947/140816 objects degraded (21.977%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":616,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35204,"data_bytes":288346600,"bytes_used":2927599616,"bytes_avail":383619457024,"bytes_total":386547056640,"degraded_objects":30947,"degraded_total":140816,"degraded_ratio":0.21976906033405291,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":218},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:49.235 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:49.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:49.911 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:49.951 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-11T05:04:49.847600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":287216104,"num_objects":35066,"num_object_clones":0,"num_object_copies":140264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150837,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2859260,"kb_used_data":563196,"kb_used_omap":14,"kb_used_meta":2295985,"kb_avail":374628100,"statfs":{"total":386547056640,"available":383619174400,"internally_reserved":0,"allocated":576712704,"data_stored":575955896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2351089206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,4,3,2,22,32,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16261120,"num_objects":-1985,"num_object_clones":0,"num_object_copies":-7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1985,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005616"},"pg_stats":[{"pgid":"1.f","version":"35'10300","reported_seq":10392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.935904+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:46.935904+0000","last_peered":"2024-09-11T05:04:46.935904+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:46.935904+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18137088,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10300,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-77}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10445","reported_seq":10560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.984323+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:44.984323+0000","last_peered":"2024-09-11T05:04:44.984323+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:44.984323+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18781344,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10443,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":35}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10020","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.344003+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:47.344003+0000","last_peered":"2024-09-11T05:04:47.344003+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:47.344003+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17529448,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10019,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-70}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10233","reported_seq":10338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.855410+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:44.855410+0000","last_peered":"2024-09-11T05:04:44.855410+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:44.855410+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17880224,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10354","reported_seq":10477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.537120+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:46.537120+0000","last_peered":"2024-09-11T05:04:46.537120+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:46.537120+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17831072,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10466","reported_seq":10571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.622872+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:46.622872+0000","last_peered":"2024-09-11T05:04:46.622872+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:46.622872+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17809408,"num_objects":2174,"num_object_clones":0,"num_object_copies":8696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10450","reported_seq":10544,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.727049+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:44.727049+0000","last_peered":"2024-09-11T05:04:44.727049+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:44.727049+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18674848,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-27}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10644","reported_seq":15501,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:47.767364+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:47.767364+0000","last_peered":"2024-09-11T05:04:47.767364+0000","last_clean":"2024-09-11T05:04:47.767364+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:47.767364+0000","last_undegraded":"2024-09-11T05:04:47.767364+0000","last_fullsized":"2024-09-11T05:04:47.767364+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17850368,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10565","reported_seq":10687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.864511+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:46.864511+0000","last_peered":"2024-09-11T05:04:46.864511+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:46.864511+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18204992,"num_objects":2223,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10562,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10386","reported_seq":10498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.498960+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:47.498960+0000","last_peered":"2024-09-11T05:04:47.498960+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:47.498960+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17544352,"num_objects":2142,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10384,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10457","reported_seq":10574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.793956+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:46.793956+0000","last_peered":"2024-09-11T05:04:46.793956+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:46.793956+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17985264,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10191","reported_seq":14792,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:47.431116+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:47.431116+0000","last_peered":"2024-09-11T05:04:47.431116+0000","last_clean":"2024-09-11T05:04:47.431116+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:47.431116+0000","last_undegraded":"2024-09-11T05:04:47.431116+0000","last_fullsized":"2024-09-11T05:04:47.431116+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17965056,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4001,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10546","reported_seq":12786,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.598146+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:47.598146+0000","last_peered":"2024-09-11T05:04:47.598146+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:47.598146+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17832528,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10545,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":921}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10411","reported_seq":10518,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.785669+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:46.785669+0000","last_peered":"2024-09-11T05:04:46.785669+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:46.785669+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10409,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-83}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10353","reported_seq":10472,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.433308+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:47.433308+0000","last_peered":"2024-09-11T05:04:47.433308+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:47.433308+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-247}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10520","reported_seq":10640,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:44.883622+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:44.883622+0000","last_peered":"2024-09-11T05:04:44.883622+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:44.883622+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486432,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10518,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-31}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":287216104,"num_objects":35066,"num_object_clones":0,"num_object_copies":140264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150837,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":574115840,"data_stored":574115840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763844,"kb_used_data":140724,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93607996,"statfs":{"total":96636764160,"available":95854587904,"internally_reserved":0,"allocated":144101376,"data_stored":143912174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,0,12,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744564,"kb_used_data":140004,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93627276,"statfs":{"total":96636764160,"available":95874330624,"internally_reserved":0,"allocated":143364096,"data_stored":143174894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,0,1,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609188,"kb_used_data":140244,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93762652,"statfs":{"total":96636764160,"available":96012955648,"internally_reserved":0,"allocated":143609856,"data_stored":143420654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480177874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,5,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741664,"kb_used_data":142224,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93630176,"statfs":{"total":96636764160,"available":95877300224,"internally_reserved":0,"allocated":145637376,"data_stored":145448174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145047552,"data_stored":145047552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142843904,"data_stored":142843904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":142876672,"data_stored":142876672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":143347712,"data_stored":143347712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:52.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:53.529 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:53.552 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-11T05:04:51.848437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285266408,"num_objects":34828,"num_object_clones":0,"num_object_copies":139312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34828,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151075,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160979,"ondisk_log_size":160979,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2873504,"kb_used_data":559384,"kb_used_omap":14,"kb_used_meta":2314033,"kb_avail":374613856,"statfs":{"total":386547056640,"available":383604588544,"internally_reserved":0,"allocated":572809216,"data_stored":572052408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2369570358},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,5,1,1,19,26,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18210816,"num_objects":-2223,"num_object_clones":0,"num_object_copies":-8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005727"},"pg_stats":[{"pgid":"1.f","version":"35'10300","reported_seq":10392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.935904+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:46.935904+0000","last_peered":"2024-09-11T05:04:46.935904+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:46.935904+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18137088,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10300,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-77}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10500","reported_seq":10614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.960790+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:49.960790+0000","last_peered":"2024-09-11T05:04:49.960790+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:49.960790+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330784,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10020","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.344003+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:47.344003+0000","last_peered":"2024-09-11T05:04:47.344003+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:47.344003+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17529448,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10019,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-70}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10295","reported_seq":10403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.926769+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:49.926769+0000","last_peered":"2024-09-11T05:04:49.926769+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:49.926769+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17372320,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10293,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10354","reported_seq":10477,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.537120+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:46.537120+0000","last_peered":"2024-09-11T05:04:46.537120+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:46.537120+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17831072,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-44}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10466","reported_seq":10571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.622872+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:46.622872+0000","last_peered":"2024-09-11T05:04:46.622872+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:46.622872+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17809408,"num_objects":2174,"num_object_clones":0,"num_object_copies":8696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10512","reported_seq":10606,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.950862+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:49.950862+0000","last_peered":"2024-09-11T05:04:49.950862+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:49.950862+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18166944,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10510,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10644","reported_seq":15501,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:47.767364+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:47.767364+0000","last_peered":"2024-09-11T05:04:47.767364+0000","last_clean":"2024-09-11T05:04:47.767364+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:47.767364+0000","last_undegraded":"2024-09-11T05:04:47.767364+0000","last_fullsized":"2024-09-11T05:04:47.767364+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17850368,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10565","reported_seq":10687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.864511+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:46.864511+0000","last_peered":"2024-09-11T05:04:46.864511+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:46.864511+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18204992,"num_objects":2223,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10562,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10386","reported_seq":10498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.498960+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:47.498960+0000","last_peered":"2024-09-11T05:04:47.498960+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:47.498960+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17544352,"num_objects":2142,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10384,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10457","reported_seq":10574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.793956+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:46.793956+0000","last_peered":"2024-09-11T05:04:46.793956+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:46.793956+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17985264,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10191","reported_seq":14792,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:47.431116+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:47.431116+0000","last_peered":"2024-09-11T05:04:47.431116+0000","last_clean":"2024-09-11T05:04:47.431116+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:47.431116+0000","last_undegraded":"2024-09-11T05:04:47.431116+0000","last_fullsized":"2024-09-11T05:04:47.431116+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17965056,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4001,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10546","reported_seq":12786,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.598146+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:47.598146+0000","last_peered":"2024-09-11T05:04:47.598146+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:47.598146+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17832528,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10545,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":921}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10411","reported_seq":10518,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:46.785669+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:46.785669+0000","last_peered":"2024-09-11T05:04:46.785669+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:46.785669+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17863840,"num_objects":2181,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10409,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-83}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10353","reported_seq":10472,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:47.433308+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:47.433308+0000","last_peered":"2024-09-11T05:04:47.433308+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:47.433308+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839840,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10351,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-247}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10579","reported_seq":10699,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.910955+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:49.910955+0000","last_peered":"2024-09-11T05:04:49.910955+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:49.910955+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18003104,"num_objects":2198,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10577,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-90}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":285266408,"num_objects":34828,"num_object_clones":0,"num_object_copies":139312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34828,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151075,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":570220544,"data_stored":570220544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160979,"ondisk_log_size":160979,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763844,"kb_used_data":140724,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93607996,"statfs":{"total":96636764160,"available":95854587904,"internally_reserved":0,"allocated":144101376,"data_stored":143912174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,0,12,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744564,"kb_used_data":140004,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93627276,"statfs":{"total":96636764160,"available":95874330624,"internally_reserved":0,"allocated":143364096,"data_stored":143174894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,0,1,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609188,"kb_used_data":140244,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93762652,"statfs":{"total":96636764160,"available":96012955648,"internally_reserved":0,"allocated":143609856,"data_stored":143420654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480177874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,5,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755908,"kb_used_data":138412,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93615932,"statfs":{"total":96636764160,"available":95862714368,"internally_reserved":0,"allocated":141733888,"data_stored":141544686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141152256,"data_stored":141152256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142843904,"data_stored":142843904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":142876672,"data_stored":142876672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":143347712,"data_stored":143347712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:53.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:54.173 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:54.194 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-11T05:04:53.849267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":279237096,"num_objects":34092,"num_object_clones":0,"num_object_copies":136368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34092,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151811,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160815,"ondisk_log_size":160815,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2870176,"kb_used_data":547736,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374617184,"statfs":{"total":386547056640,"available":383607996416,"internally_reserved":0,"allocated":560881664,"data_stored":560124856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2378090038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5,4,12,25,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22241280,"num_objects":-2715,"num_object_clones":0,"num_object_copies":-10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2715,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005911"},"pg_stats":[{"pgid":"1.f","version":"35'10363","reported_seq":10458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.155492+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:52.155492+0000","last_peered":"2024-09-11T05:04:52.155492+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:52.155492+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17620992,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10363,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10500","reported_seq":10614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.960790+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:49.960790+0000","last_peered":"2024-09-11T05:04:49.960790+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:49.960790+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330784,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10078","reported_seq":10183,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.643381+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:52.643381+0000","last_peered":"2024-09-11T05:04:52.643381+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:52.643381+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17054312,"num_objects":2083,"num_object_clones":0,"num_object_copies":8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10077,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10295","reported_seq":10403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.926769+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:49.926769+0000","last_peered":"2024-09-11T05:04:49.926769+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:49.926769+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17372320,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10293,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10414","reported_seq":10537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.045261+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:52.045261+0000","last_peered":"2024-09-11T05:04:52.045261+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:52.045261+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17339552,"num_objects":2117,"num_object_clones":0,"num_object_copies":8468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10412,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10529","reported_seq":10635,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.041937+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:52.041937+0000","last_peered":"2024-09-11T05:04:52.041937+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:52.041937+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17293312,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10529,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10512","reported_seq":10606,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.950862+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:49.950862+0000","last_peered":"2024-09-11T05:04:49.950862+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:49.950862+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18166944,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10510,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10711","reported_seq":15568,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:52.836270+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:52.836270+0000","last_peered":"2024-09-11T05:04:52.836270+0000","last_clean":"2024-09-11T05:04:52.836270+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:52.836270+0000","last_undegraded":"2024-09-11T05:04:52.836270+0000","last_fullsized":"2024-09-11T05:04:52.836270+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17301504,"num_objects":2112,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10628","reported_seq":10748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.042403+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:52.042403+0000","last_peered":"2024-09-11T05:04:52.042403+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:52.042403+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17688896,"num_objects":2160,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10448","reported_seq":10560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.597376+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:52.597376+0000","last_peered":"2024-09-11T05:04:52.597376+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:52.597376+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17036448,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10446,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-226}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10526","reported_seq":10643,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.155037+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:52.155037+0000","last_peered":"2024-09-11T05:04:52.155037+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:52.155037+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17420016,"num_objects":2127,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10244","reported_seq":14845,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:52.785813+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:52.785813+0000","last_peered":"2024-09-11T05:04:52.785813+0000","last_clean":"2024-09-11T05:04:52.785813+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:52.785813+0000","last_undegraded":"2024-09-11T05:04:52.785813+0000","last_fullsized":"2024-09-11T05:04:52.785813+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17530880,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4054,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10597","reported_seq":12929,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.688536+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:52.688536+0000","last_peered":"2024-09-11T05:04:52.688536+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:52.688536+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17414736,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10596,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10485","reported_seq":10592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.065727+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:52.065727+0000","last_peered":"2024-09-11T05:04:52.065727+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:52.065727+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17257632,"num_objects":2107,"num_object_clones":0,"num_object_copies":8428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-157}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10406","reported_seq":10525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.470902+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:52.470902+0000","last_peered":"2024-09-11T05:04:52.470902+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:52.470902+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16405664,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10404,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10579","reported_seq":10699,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.910955+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:49.910955+0000","last_peered":"2024-09-11T05:04:49.910955+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:49.910955+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18003104,"num_objects":2198,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10577,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-90}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":279237096,"num_objects":34092,"num_object_clones":0,"num_object_copies":136368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34092,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151811,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":558104576,"data_stored":558104576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160815,"ondisk_log_size":160815,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764140,"kb_used_data":136860,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93607700,"statfs":{"total":96636764160,"available":95854284800,"internally_reserved":0,"allocated":140144640,"data_stored":139955438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,6,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744832,"kb_used_data":136112,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93627008,"statfs":{"total":96636764160,"available":95874056192,"internally_reserved":0,"allocated":139378688,"data_stored":139189486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,2,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605296,"kb_used_data":136352,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93766544,"statfs":{"total":96636764160,"available":96016941056,"internally_reserved":0,"allocated":139624448,"data_stored":139435246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480177874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,3,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755908,"kb_used_data":138412,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93615932,"statfs":{"total":96636764160,"available":95862714368,"internally_reserved":0,"allocated":141733888,"data_stored":141544686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141152256,"data_stored":141152256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":138764288,"data_stored":138764288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138891264,"data_stored":138891264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139296768,"data_stored":139296768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:54.195 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:54.195 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10363", 'reported_seq': 10458, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.155492+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:52.155492+0000', 'last_peered': '2024-09-11T05:04:52.155492+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:52.155492+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17620992, 'num_objects': 2151, 'num_object_clones': 0, 'num_object_copies': 8604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10363, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -140}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10500", 'reported_seq': 10614, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:49.960790+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:49.960790+0000', 'last_peered': '2024-09-11T05:04:49.960790+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:49.960790+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18330784, 'num_objects': 2238, 'num_object_clones': 0, 'num_object_copies': 8952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10498, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -20}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10078", 'reported_seq': 10183, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.643381+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:52.643381+0000', 'last_peered': '2024-09-11T05:04:52.643381+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:52.643381+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17054312, 'num_objects': 2083, 'num_object_clones': 0, 'num_object_copies': 8332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2083, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10077, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -128}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10295", 'reported_seq': 10403, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:49.926769+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:49.926769+0000', 'last_peered': '2024-09-11T05:04:49.926769+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:49.926769+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17372320, 'num_objects': 2121, 'num_object_clones': 0, 'num_object_copies': 8484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10293, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10414", 'reported_seq': 10537, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.045261+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:52.045261+0000', 'last_peered': '2024-09-11T05:04:52.045261+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:52.045261+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17339552, 'num_objects': 2117, 'num_object_clones': 0, 'num_object_copies': 8468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10412, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -104}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:54.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10529", 'reported_seq': 10635, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.041937+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:52.041937+0000', 'last_peered': '2024-09-11T05:04:52.041937+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:52.041937+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17293312, 'num_objects': 2111, 'num_object_clones': 0, 'num_object_copies': 8444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10529, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10512", 'reported_seq': 10606, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:49.950862+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:49.950862+0000', 'last_peered': '2024-09-11T05:04:49.950862+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:49.950862+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18166944, 'num_objects': 2218, 'num_object_clones': 0, 'num_object_copies': 8872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10510, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -89}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10628", 'reported_seq': 10748, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.042403+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:52.042403+0000', 'last_peered': '2024-09-11T05:04:52.042403+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:52.042403+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17688896, 'num_objects': 2160, 'num_object_clones': 0, 'num_object_copies': 8640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10625, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -206}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10448", 'reported_seq': 10560, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.597376+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:52.597376+0000', 'last_peered': '2024-09-11T05:04:52.597376+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:52.597376+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17036448, 'num_objects': 2080, 'num_object_clones': 0, 'num_object_copies': 8320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10446, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -226}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10526", 'reported_seq': 10643, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.155037+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:52.155037+0000', 'last_peered': '2024-09-11T05:04:52.155037+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:52.155037+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17420016, 'num_objects': 2127, 'num_object_clones': 0, 'num_object_copies': 8508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10523, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -171}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10597", 'reported_seq': 12929, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.688536+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:52.688536+0000', 'last_peered': '2024-09-11T05:04:52.688536+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:52.688536+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17414736, 'num_objects': 2126, 'num_object_clones': 0, 'num_object_copies': 8504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10596, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1124, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 916}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:54.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:54.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10485", 'reported_seq': 10592, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.065727+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:52.065727+0000', 'last_peered': '2024-09-11T05:04:52.065727+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:52.065727+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17257632, 'num_objects': 2107, 'num_object_clones': 0, 'num_object_copies': 8428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10483, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:54.198 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:54.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10406", 'reported_seq': 10525, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:52.470902+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:52.470902+0000', 'last_peered': '2024-09-11T05:04:52.470902+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:52.470902+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16405664, 'num_objects': 2003, 'num_object_clones': 0, 'num_object_copies': 8012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10404, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -300}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:54.198 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:54.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10579", 'reported_seq': 10699, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:49.910955+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:49.910955+0000', 'last_peered': '2024-09-11T05:04:49.910955+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:49.910955+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18003104, 'num_objects': 2198, 'num_object_clones': 0, 'num_object_copies': 8792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10577, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -90}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:54.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:04:55.073 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:55.074 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30901/136368 objects degraded (22.660%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":622,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":34092,"data_bytes":279237096,"bytes_used":2939060224,"bytes_avail":383607996416,"bytes_total":386547056640,"degraded_objects":30901,"degraded_total":136368,"degraded_ratio":0.22660008213070515,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":226},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:04:55.103 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:04:55.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:55.644 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:55.667 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-11T05:04:53.849267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":279237096,"num_objects":34092,"num_object_clones":0,"num_object_copies":136368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34092,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151811,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160815,"ondisk_log_size":160815,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2870176,"kb_used_data":547736,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374617184,"statfs":{"total":386547056640,"available":383607996416,"internally_reserved":0,"allocated":560881664,"data_stored":560124856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2378090038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5,4,12,25,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22241280,"num_objects":-2715,"num_object_clones":0,"num_object_copies":-10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2715,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005911"},"pg_stats":[{"pgid":"1.f","version":"35'10363","reported_seq":10458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.155492+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:52.155492+0000","last_peered":"2024-09-11T05:04:52.155492+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:52.155492+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17620992,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10363,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10500","reported_seq":10614,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.960790+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:49.960790+0000","last_peered":"2024-09-11T05:04:49.960790+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:49.960790+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330784,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10078","reported_seq":10183,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.643381+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:52.643381+0000","last_peered":"2024-09-11T05:04:52.643381+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:52.643381+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17054312,"num_objects":2083,"num_object_clones":0,"num_object_copies":8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10077,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10295","reported_seq":10403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.926769+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:49.926769+0000","last_peered":"2024-09-11T05:04:49.926769+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:49.926769+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17372320,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10293,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10414","reported_seq":10537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.045261+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:52.045261+0000","last_peered":"2024-09-11T05:04:52.045261+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:52.045261+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17339552,"num_objects":2117,"num_object_clones":0,"num_object_copies":8468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10412,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10529","reported_seq":10635,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.041937+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:52.041937+0000","last_peered":"2024-09-11T05:04:52.041937+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:52.041937+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17293312,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10529,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10512","reported_seq":10606,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.950862+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:49.950862+0000","last_peered":"2024-09-11T05:04:49.950862+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:49.950862+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18166944,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10510,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10711","reported_seq":15568,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:52.836270+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:52.836270+0000","last_peered":"2024-09-11T05:04:52.836270+0000","last_clean":"2024-09-11T05:04:52.836270+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:52.836270+0000","last_undegraded":"2024-09-11T05:04:52.836270+0000","last_fullsized":"2024-09-11T05:04:52.836270+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17301504,"num_objects":2112,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10628","reported_seq":10748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.042403+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:52.042403+0000","last_peered":"2024-09-11T05:04:52.042403+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:52.042403+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17688896,"num_objects":2160,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10448","reported_seq":10560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.597376+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:52.597376+0000","last_peered":"2024-09-11T05:04:52.597376+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:52.597376+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17036448,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10446,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-226}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10526","reported_seq":10643,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.155037+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:52.155037+0000","last_peered":"2024-09-11T05:04:52.155037+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:52.155037+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17420016,"num_objects":2127,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10244","reported_seq":14845,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:52.785813+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:52.785813+0000","last_peered":"2024-09-11T05:04:52.785813+0000","last_clean":"2024-09-11T05:04:52.785813+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:52.785813+0000","last_undegraded":"2024-09-11T05:04:52.785813+0000","last_fullsized":"2024-09-11T05:04:52.785813+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17530880,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4054,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10597","reported_seq":12929,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.688536+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:52.688536+0000","last_peered":"2024-09-11T05:04:52.688536+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:52.688536+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17414736,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10596,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10485","reported_seq":10592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.065727+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:52.065727+0000","last_peered":"2024-09-11T05:04:52.065727+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:52.065727+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17257632,"num_objects":2107,"num_object_clones":0,"num_object_copies":8428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-157}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10406","reported_seq":10525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:52.470902+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:52.470902+0000","last_peered":"2024-09-11T05:04:52.470902+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:52.470902+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16405664,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10404,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10579","reported_seq":10699,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:49.910955+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:49.910955+0000","last_peered":"2024-09-11T05:04:49.910955+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:49.910955+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18003104,"num_objects":2198,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10577,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-90}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":279237096,"num_objects":34092,"num_object_clones":0,"num_object_copies":136368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34092,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151811,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":558104576,"data_stored":558104576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160815,"ondisk_log_size":160815,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764140,"kb_used_data":136860,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93607700,"statfs":{"total":96636764160,"available":95854284800,"internally_reserved":0,"allocated":140144640,"data_stored":139955438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,6,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744832,"kb_used_data":136112,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93627008,"statfs":{"total":96636764160,"available":95874056192,"internally_reserved":0,"allocated":139378688,"data_stored":139189486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,2,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605296,"kb_used_data":136352,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93766544,"statfs":{"total":96636764160,"available":96016941056,"internally_reserved":0,"allocated":139624448,"data_stored":139435246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480177874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,3,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755908,"kb_used_data":138412,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93615932,"statfs":{"total":96636764160,"available":95862714368,"internally_reserved":0,"allocated":141733888,"data_stored":141544686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141152256,"data_stored":141152256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":138764288,"data_stored":138764288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138891264,"data_stored":138891264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139296768,"data_stored":139296768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:58.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:59.298 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:59.299 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:59.324 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-11T05:04:57.850998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272298472,"num_objects":33245,"num_object_clones":0,"num_object_copies":132980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152658,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160962,"ondisk_log_size":160962,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2858544,"kb_used_data":536100,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374628816,"statfs":{"total":386547056640,"available":383619907584,"internally_reserved":0,"allocated":548966400,"data_stored":548209592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2378090038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,10,6,11,17,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20914176,"num_objects":-2553,"num_object_clones":0,"num_object_copies":-10212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005314"},"pg_stats":[{"pgid":"1.f","version":"35'10424","reported_seq":10519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.079293+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:57.079293+0000","last_peered":"2024-09-11T05:04:57.079293+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:57.079293+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17121280,"num_objects":2090,"num_object_clones":0,"num_object_copies":8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10553","reported_seq":10668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.613554+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:54.613554+0000","last_peered":"2024-09-11T05:04:54.613554+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:54.613554+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896608,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10551,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-73}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10125","reported_seq":10230,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.602235+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:57.602235+0000","last_peered":"2024-09-11T05:04:57.602235+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:57.602235+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16669288,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10124,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-175}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10357","reported_seq":10465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.687782+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:54.687782+0000","last_peered":"2024-09-11T05:04:54.687782+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:54.687782+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16864416,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-202}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10482","reported_seq":10605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.208386+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:57.208386+0000","last_peered":"2024-09-11T05:04:57.208386+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:57.208386+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782496,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10480,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10586","reported_seq":10692,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.220797+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:57.220797+0000","last_peered":"2024-09-11T05:04:57.220797+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:57.220797+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16826368,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10586,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10575","reported_seq":10668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.965192+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:54.965192+0000","last_peered":"2024-09-11T05:04:54.965192+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:54.965192+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17650848,"num_objects":2155,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10573,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-152}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10711","reported_seq":15568,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:52.836270+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:52.836270+0000","last_peered":"2024-09-11T05:04:52.836270+0000","last_clean":"2024-09-11T05:04:52.836270+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:52.836270+0000","last_undegraded":"2024-09-11T05:04:52.836270+0000","last_fullsized":"2024-09-11T05:04:52.836270+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17301504,"num_objects":2112,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10685","reported_seq":10808,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.121350+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:57.121350+0000","last_peered":"2024-09-11T05:04:57.121350+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:57.121350+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17221952,"num_objects":2103,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10485","reported_seq":10597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.638920+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:57.638920+0000","last_peered":"2024-09-11T05:04:57.638920+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:57.638920+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16733344,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10593","reported_seq":10710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.130909+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:57.130909+0000","last_peered":"2024-09-11T05:04:57.130909+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:57.130909+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16871152,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10244","reported_seq":14845,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:52.785813+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:52.785813+0000","last_peered":"2024-09-11T05:04:52.785813+0000","last_clean":"2024-09-11T05:04:52.785813+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:52.785813+0000","last_undegraded":"2024-09-11T05:04:52.785813+0000","last_fullsized":"2024-09-11T05:04:52.785813+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17530880,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4054,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10672","reported_seq":13135,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.726555+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:57.726555+0000","last_peered":"2024-09-11T05:04:57.726555+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:57.726555+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16800336,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10671,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10548","reported_seq":10655,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.010971+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:57.010971+0000","last_peered":"2024-09-11T05:04:57.010971+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:57.010971+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16741536,"num_objects":2044,"num_object_clones":0,"num_object_copies":8176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10474","reported_seq":10592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.700238+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:57.700238+0000","last_peered":"2024-09-11T05:04:57.700238+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:57.700238+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15848608,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10472,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10648","reported_seq":10768,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.861595+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:54.861595+0000","last_peered":"2024-09-11T05:04:54.861595+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:54.861595+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17437856,"num_objects":2129,"num_object_clones":0,"num_object_copies":8516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10646,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":272298472,"num_objects":33245,"num_object_clones":0,"num_object_copies":132980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152658,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":546070528,"data_stored":546070528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160962,"ondisk_log_size":160962,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764396,"kb_used_data":132952,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93607444,"statfs":{"total":96636764160,"available":95854022656,"internally_reserved":0,"allocated":136142848,"data_stored":135953646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745176,"kb_used_data":132292,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93626664,"statfs":{"total":96636764160,"available":95873703936,"internally_reserved":0,"allocated":135467008,"data_stored":135277806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605296,"kb_used_data":136352,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93766544,"statfs":{"total":96636764160,"available":96016941056,"internally_reserved":0,"allocated":139624448,"data_stored":139435246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480177874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,3,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743676,"kb_used_data":134504,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93628164,"statfs":{"total":96636764160,"available":95875239936,"internally_reserved":0,"allocated":137732096,"data_stored":137542894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":137113600,"data_stored":137113600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":138764288,"data_stored":138764288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134909952,"data_stored":134909952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135282688,"data_stored":135282688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:59.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:04:59.964 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:04:59.965 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:04:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-11T05:04:59.851683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":271282664,"num_objects":33121,"num_object_clones":0,"num_object_copies":132484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152782,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2858824,"kb_used_data":532220,"kb_used_omap":14,"kb_used_meta":2326513,"kb_avail":374628536,"statfs":{"total":386547056640,"available":383619620864,"internally_reserved":0,"allocated":544993280,"data_stored":544236472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2382349878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,10,11,13,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17063936,"num_objects":-2083,"num_object_clones":0,"num_object_copies":-8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2083,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004943"},"pg_stats":[{"pgid":"1.f","version":"35'10424","reported_seq":10519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.079293+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:57.079293+0000","last_peered":"2024-09-11T05:04:57.079293+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:57.079293+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17121280,"num_objects":2090,"num_object_clones":0,"num_object_copies":8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10553","reported_seq":10668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.613554+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:54.613554+0000","last_peered":"2024-09-11T05:04:54.613554+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:54.613554+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896608,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10551,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-73}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10125","reported_seq":10230,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.602235+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:57.602235+0000","last_peered":"2024-09-11T05:04:57.602235+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:57.602235+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16669288,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10124,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-175}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10357","reported_seq":10465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.687782+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:54.687782+0000","last_peered":"2024-09-11T05:04:54.687782+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:54.687782+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16864416,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-202}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10482","reported_seq":10605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.208386+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:57.208386+0000","last_peered":"2024-09-11T05:04:57.208386+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:57.208386+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782496,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10480,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10586","reported_seq":10692,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.220797+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:57.220797+0000","last_peered":"2024-09-11T05:04:57.220797+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:57.220797+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16826368,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10586,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10575","reported_seq":10668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.965192+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:54.965192+0000","last_peered":"2024-09-11T05:04:54.965192+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:54.965192+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17650848,"num_objects":2155,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10573,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-152}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10775","reported_seq":15631,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:58.110921+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:58.110921+0000","last_peered":"2024-09-11T05:04:58.110921+0000","last_clean":"2024-09-11T05:04:58.110921+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:58.110921+0000","last_undegraded":"2024-09-11T05:04:58.110921+0000","last_fullsized":"2024-09-11T05:04:58.110921+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1485,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10685","reported_seq":10808,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.121350+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:57.121350+0000","last_peered":"2024-09-11T05:04:57.121350+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:57.121350+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17221952,"num_objects":2103,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10485","reported_seq":10597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.638920+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:57.638920+0000","last_peered":"2024-09-11T05:04:57.638920+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:57.638920+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16733344,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10593","reported_seq":10710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.130909+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:57.130909+0000","last_peered":"2024-09-11T05:04:57.130909+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:57.130909+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16871152,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10304","reported_seq":14905,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:57.939691+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:57.939691+0000","last_peered":"2024-09-11T05:04:57.939691+0000","last_clean":"2024-09-11T05:04:57.939691+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:57.939691+0000","last_undegraded":"2024-09-11T05:04:57.939691+0000","last_fullsized":"2024-09-11T05:04:57.939691+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039360,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4114,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10672","reported_seq":13135,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.726555+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:57.726555+0000","last_peered":"2024-09-11T05:04:57.726555+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:57.726555+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16800336,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10671,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10548","reported_seq":10655,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.010971+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:57.010971+0000","last_peered":"2024-09-11T05:04:57.010971+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:57.010971+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16741536,"num_objects":2044,"num_object_clones":0,"num_object_copies":8176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10474","reported_seq":10592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.700238+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:57.700238+0000","last_peered":"2024-09-11T05:04:57.700238+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:57.700238+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15848608,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10472,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10648","reported_seq":10768,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.861595+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:54.861595+0000","last_peered":"2024-09-11T05:04:54.861595+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:54.861595+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17437856,"num_objects":2129,"num_object_clones":0,"num_object_copies":8516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10646,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":271282664,"num_objects":33121,"num_object_clones":0,"num_object_copies":132484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152782,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":542019584,"data_stored":542019584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764396,"kb_used_data":132952,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93607444,"statfs":{"total":96636764160,"available":95854022656,"internally_reserved":0,"allocated":136142848,"data_stored":135953646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745176,"kb_used_data":132292,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93626664,"statfs":{"total":96636764160,"available":95873703936,"internally_reserved":0,"allocated":135467008,"data_stored":135277806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605576,"kb_used_data":132472,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93766264,"statfs":{"total":96636764160,"available":96016654336,"internally_reserved":0,"allocated":135651328,"data_stored":135462126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484437714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743676,"kb_used_data":134504,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93628164,"statfs":{"total":96636764160,"available":95875239936,"internally_reserved":0,"allocated":137732096,"data_stored":137542894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":137113600,"data_stored":137113600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134713344,"data_stored":134713344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134909952,"data_stored":134909952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135282688,"data_stored":135282688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:04:59.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:04:59.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10424", 'reported_seq': 10519, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.079293+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:04:57.079293+0000', 'last_peered': '2024-09-11T05:04:57.079293+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:04:57.079293+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17121280, 'num_objects': 2090, 'num_object_clones': 0, 'num_object_copies': 8360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10424, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:59.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:04:59.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10553", 'reported_seq': 10668, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:54.613554+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:04:54.613554+0000', 'last_peered': '2024-09-11T05:04:54.613554+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:04:54.613554+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17896608, 'num_objects': 2185, 'num_object_clones': 0, 'num_object_copies': 8740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10551, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -73}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10125", 'reported_seq': 10230, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.602235+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:04:57.602235+0000', 'last_peered': '2024-09-11T05:04:57.602235+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:04:57.602235+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16669288, 'num_objects': 2036, 'num_object_clones': 0, 'num_object_copies': 8144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2036, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10124, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -175}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10357", 'reported_seq': 10465, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:54.687782+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:04:54.687782+0000', 'last_peered': '2024-09-11T05:04:54.687782+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:04:54.687782+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16864416, 'num_objects': 2059, 'num_object_clones': 0, 'num_object_copies': 8236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10355, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10482", 'reported_seq': 10605, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.208386+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:04:57.208386+0000', 'last_peered': '2024-09-11T05:04:57.208386+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:04:57.208386+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16782496, 'num_objects': 2049, 'num_object_clones': 0, 'num_object_copies': 8196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10480, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -172}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10586", 'reported_seq': 10692, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.220797+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:04:57.220797+0000', 'last_peered': '2024-09-11T05:04:57.220797+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:04:57.220797+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16826368, 'num_objects': 2054, 'num_object_clones': 0, 'num_object_copies': 8216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10586, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10575", 'reported_seq': 10668, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:54.965192+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:04:54.965192+0000', 'last_peered': '2024-09-11T05:04:54.965192+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:04:54.965192+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17650848, 'num_objects': 2155, 'num_object_clones': 0, 'num_object_copies': 8620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10573, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -152}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:59.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10685", 'reported_seq': 10808, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.121350+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:04:57.121350+0000', 'last_peered': '2024-09-11T05:04:57.121350+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:04:57.121350+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17221952, 'num_objects': 2103, 'num_object_clones': 0, 'num_object_copies': 8412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10682, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10485", 'reported_seq': 10597, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.638920+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:04:57.638920+0000', 'last_peered': '2024-09-11T05:04:57.638920+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:04:57.638920+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16733344, 'num_objects': 2043, 'num_object_clones': 0, 'num_object_copies': 8172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10483, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -263}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10593", 'reported_seq': 10710, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.130909+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:04:57.130909+0000', 'last_peered': '2024-09-11T05:04:57.130909+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:04:57.130909+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16871152, 'num_objects': 2060, 'num_object_clones': 0, 'num_object_copies': 8240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10590, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -238}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10672", 'reported_seq': 13135, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.726555+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:04:57.726555+0000', 'last_peered': '2024-09-11T05:04:57.726555+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:04:57.726555+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16800336, 'num_objects': 2051, 'num_object_clones': 0, 'num_object_copies': 8204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1144, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10671, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1190, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10548", 'reported_seq': 10655, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.010971+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:04:57.010971+0000', 'last_peered': '2024-09-11T05:04:57.010971+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:04:57.010971+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16741536, 'num_objects': 2044, 'num_object_clones': 0, 'num_object_copies': 8176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10546, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:04:59.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:04:59.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10474", 'reported_seq': 10592, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:57.700238+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:04:57.700238+0000', 'last_peered': '2024-09-11T05:04:57.700238+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:04:57.700238+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15848608, 'num_objects': 1935, 'num_object_clones': 0, 'num_object_copies': 7740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10472, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -368}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:04:59.989 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:04:59.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10648", 'reported_seq': 10768, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:04:54.861595+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:04:54.861595+0000', 'last_peered': '2024-09-11T05:04:54.861595+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:04:54.861595+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17437856, 'num_objects': 2129, 'num_object_clones': 0, 'num_object_copies': 8516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10646, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:04:59.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:00.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:00.771 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30835/132484 objects degraded (23.275%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":627,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33121,"data_bytes":271282664,"bytes_used":2927435776,"bytes_avail":383619620864,"bytes_total":386547056640,"degraded_objects":30835,"degraded_total":132484,"degraded_ratio":0.23274508619908818,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":173},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:00.795 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:00.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:01.486 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-11T05:04:59.851683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":271282664,"num_objects":33121,"num_object_clones":0,"num_object_copies":132484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152782,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2858824,"kb_used_data":532220,"kb_used_omap":14,"kb_used_meta":2326513,"kb_avail":374628536,"statfs":{"total":386547056640,"available":383619620864,"internally_reserved":0,"allocated":544993280,"data_stored":544236472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2382349878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,10,11,13,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17063936,"num_objects":-2083,"num_object_clones":0,"num_object_copies":-8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2083,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004943"},"pg_stats":[{"pgid":"1.f","version":"35'10424","reported_seq":10519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.079293+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:04:57.079293+0000","last_peered":"2024-09-11T05:04:57.079293+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:04:57.079293+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17121280,"num_objects":2090,"num_object_clones":0,"num_object_copies":8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10553","reported_seq":10668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.613554+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:04:54.613554+0000","last_peered":"2024-09-11T05:04:54.613554+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:04:54.613554+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896608,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10551,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-73}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10125","reported_seq":10230,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.602235+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:04:57.602235+0000","last_peered":"2024-09-11T05:04:57.602235+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:04:57.602235+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16669288,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10124,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-175}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10357","reported_seq":10465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.687782+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:04:54.687782+0000","last_peered":"2024-09-11T05:04:54.687782+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:04:54.687782+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16864416,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-202}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10482","reported_seq":10605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.208386+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:04:57.208386+0000","last_peered":"2024-09-11T05:04:57.208386+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:04:57.208386+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782496,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10480,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10586","reported_seq":10692,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.220797+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:04:57.220797+0000","last_peered":"2024-09-11T05:04:57.220797+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:04:57.220797+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16826368,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10586,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10575","reported_seq":10668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.965192+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:04:54.965192+0000","last_peered":"2024-09-11T05:04:54.965192+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:04:54.965192+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17650848,"num_objects":2155,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10573,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-152}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10775","reported_seq":15631,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:58.110921+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:04:58.110921+0000","last_peered":"2024-09-11T05:04:58.110921+0000","last_clean":"2024-09-11T05:04:58.110921+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:04:58.110921+0000","last_undegraded":"2024-09-11T05:04:58.110921+0000","last_fullsized":"2024-09-11T05:04:58.110921+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1485,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10685","reported_seq":10808,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.121350+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:04:57.121350+0000","last_peered":"2024-09-11T05:04:57.121350+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:04:57.121350+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17221952,"num_objects":2103,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10485","reported_seq":10597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.638920+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:04:57.638920+0000","last_peered":"2024-09-11T05:04:57.638920+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:04:57.638920+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16733344,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10593","reported_seq":10710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.130909+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:04:57.130909+0000","last_peered":"2024-09-11T05:04:57.130909+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:04:57.130909+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16871152,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10304","reported_seq":14905,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:04:57.939691+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:04:57.939691+0000","last_peered":"2024-09-11T05:04:57.939691+0000","last_clean":"2024-09-11T05:04:57.939691+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:04:57.939691+0000","last_undegraded":"2024-09-11T05:04:57.939691+0000","last_fullsized":"2024-09-11T05:04:57.939691+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039360,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4114,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10672","reported_seq":13135,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.726555+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:04:57.726555+0000","last_peered":"2024-09-11T05:04:57.726555+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:04:57.726555+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16800336,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10671,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10548","reported_seq":10655,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.010971+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:04:57.010971+0000","last_peered":"2024-09-11T05:04:57.010971+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:04:57.010971+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16741536,"num_objects":2044,"num_object_clones":0,"num_object_copies":8176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10474","reported_seq":10592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:57.700238+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:04:57.700238+0000","last_peered":"2024-09-11T05:04:57.700238+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:04:57.700238+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15848608,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10472,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-368}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10648","reported_seq":10768,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:04:54.861595+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:04:54.861595+0000","last_peered":"2024-09-11T05:04:54.861595+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:04:54.861595+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17437856,"num_objects":2129,"num_object_clones":0,"num_object_copies":8516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10646,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":271282664,"num_objects":33121,"num_object_clones":0,"num_object_copies":132484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152782,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":542019584,"data_stored":542019584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764396,"kb_used_data":132952,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93607444,"statfs":{"total":96636764160,"available":95854022656,"internally_reserved":0,"allocated":136142848,"data_stored":135953646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745176,"kb_used_data":132292,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93626664,"statfs":{"total":96636764160,"available":95873703936,"internally_reserved":0,"allocated":135467008,"data_stored":135277806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605576,"kb_used_data":132472,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93766264,"statfs":{"total":96636764160,"available":96016654336,"internally_reserved":0,"allocated":135651328,"data_stored":135462126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484437714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743676,"kb_used_data":134504,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93628164,"statfs":{"total":96636764160,"available":95875239936,"internally_reserved":0,"allocated":137732096,"data_stored":137542894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":137113600,"data_stored":137113600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134713344,"data_stored":134713344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134909952,"data_stored":134909952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135282688,"data_stored":135282688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:04.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:05.186 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:05.186 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:05.207 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-11T05:05:03.853028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":263336424,"num_objects":32151,"num_object_clones":0,"num_object_copies":128604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153752,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839344,"kb_used_data":516880,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374648016,"statfs":{"total":386547056640,"available":383639568384,"internally_reserved":0,"allocated":529285120,"data_stored":528528312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2378090038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,0,17,18,15,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21929984,"num_objects":-2677,"num_object_clones":0,"num_object_copies":-10708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2677,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004591"},"pg_stats":[{"pgid":"1.f","version":"35'10477","reported_seq":10572,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.279695+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:02.279695+0000","last_peered":"2024-09-11T05:05:02.279695+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:02.279695+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16687104,"num_objects":2037,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10625","reported_seq":10739,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:00.173855+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:00.173855+0000","last_peered":"2024-09-11T05:05:00.173855+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:00.173855+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17306784,"num_objects":2113,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10179","reported_seq":10284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.650335+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:02.650335+0000","last_peered":"2024-09-11T05:05:02.650335+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:02.650335+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16226920,"num_objects":1982,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10178,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-229}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10431","reported_seq":10539,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:00.104842+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:00.104842+0000","last_peered":"2024-09-11T05:05:00.104842+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:00.104842+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16258208,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10429,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10543","reported_seq":10666,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.303956+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:02.303956+0000","last_peered":"2024-09-11T05:05:02.303956+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:02.303956+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282784,"num_objects":1988,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10653","reported_seq":10759,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.134041+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:02.134041+0000","last_peered":"2024-09-11T05:05:02.134041+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:02.134041+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16277504,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10640","reported_seq":10734,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:00.099243+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:00.099243+0000","last_peered":"2024-09-11T05:05:00.099243+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:00.099243+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17118368,"num_objects":2090,"num_object_clones":0,"num_object_copies":8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-217}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10829","reported_seq":15687,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:03.207127+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:03.207127+0000","last_peered":"2024-09-11T05:05:03.207127+0000","last_clean":"2024-09-11T05:05:03.207127+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:03.207127+0000","last_undegraded":"2024-09-11T05:05:03.207127+0000","last_fullsized":"2024-09-11T05:05:03.207127+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16334848,"num_objects":1994,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1539,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10745","reported_seq":10867,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.258848+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:02.258848+0000","last_peered":"2024-09-11T05:05:02.258848+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:02.258848+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16730432,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10553","reported_seq":10665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.741676+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:02.741676+0000","last_peered":"2024-09-11T05:05:02.741676+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:02.741676+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16176288,"num_objects":1975,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10551,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10658","reported_seq":10775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.247599+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:02.247599+0000","last_peered":"2024-09-11T05:05:02.247599+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:02.247599+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338672,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10655,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10368","reported_seq":14969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:02.837089+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:02.837089+0000","last_peered":"2024-09-11T05:05:02.837089+0000","last_clean":"2024-09-11T05:05:02.837089+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:02.837089+0000","last_undegraded":"2024-09-11T05:05:02.837089+0000","last_fullsized":"2024-09-11T05:05:02.837089+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":2016,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10716","reported_seq":13281,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.812762+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:02.812762+0000","last_peered":"2024-09-11T05:05:02.812762+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:02.812762+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16439888,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10715,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":914}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10609","reported_seq":10716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:01.962299+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:01.962299+0000","last_peered":"2024-09-11T05:05:01.962299+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:01.962299+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16241824,"num_objects":1983,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10607,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10525","reported_seq":10644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.720821+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:02.720821+0000","last_peered":"2024-09-11T05:05:02.720821+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:02.720821+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10705","reported_seq":10825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:00.091672+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:00.091672+0000","last_peered":"2024-09-11T05:05:00.091672+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:00.091672+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16970912,"num_objects":2072,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":263336424,"num_objects":32151,"num_object_clones":0,"num_object_copies":128604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153752,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":525963264,"data_stored":525963264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764740,"kb_used_data":129140,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93607100,"statfs":{"total":96636764160,"available":95853670400,"internally_reserved":0,"allocated":132239360,"data_stored":132050158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,0,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745568,"kb_used_data":128496,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93626272,"statfs":{"total":96636764160,"available":95873302528,"internally_reserved":0,"allocated":131579904,"data_stored":131390702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589224,"kb_used_data":128600,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93782616,"statfs":{"total":96636764160,"available":96033398784,"internally_reserved":0,"allocated":131686400,"data_stored":131497198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471658194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739812,"kb_used_data":130644,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93632028,"statfs":{"total":96636764160,"available":95879196672,"internally_reserved":0,"allocated":133779456,"data_stored":133590254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":133046272,"data_stored":133046272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130654208,"data_stored":130654208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130928640,"data_stored":130928640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":131334144,"data_stored":131334144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:05.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:05.859 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:05.887 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-11T05:05:05.853550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":261149160,"num_objects":31884,"num_object_clones":0,"num_object_copies":127536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154019,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839648,"kb_used_data":513016,"kb_used_omap":14,"kb_used_meta":2326513,"kb_avail":374647712,"statfs":{"total":386547056640,"available":383639257088,"internally_reserved":0,"allocated":525328384,"data_stored":524571576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2382349878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,0,16,16,14,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18087936,"num_objects":-2208,"num_object_clones":0,"num_object_copies":-8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004282"},"pg_stats":[{"pgid":"1.f","version":"35'10477","reported_seq":10572,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.279695+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:02.279695+0000","last_peered":"2024-09-11T05:05:02.279695+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:02.279695+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16687104,"num_objects":2037,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10702","reported_seq":10813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.144031+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:05.144031+0000","last_peered":"2024-09-11T05:05:05.144031+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:05.144031+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16676000,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10179","reported_seq":10284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.650335+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:02.650335+0000","last_peered":"2024-09-11T05:05:02.650335+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:02.650335+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16226920,"num_objects":1982,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10178,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-229}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10495","reported_seq":10603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.160527+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:05.160527+0000","last_peered":"2024-09-11T05:05:05.160527+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:05.160527+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15733920,"num_objects":1921,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10543","reported_seq":10666,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.303956+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:02.303956+0000","last_peered":"2024-09-11T05:05:02.303956+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:02.303956+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282784,"num_objects":1988,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10653","reported_seq":10759,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.134041+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:02.134041+0000","last_peered":"2024-09-11T05:05:02.134041+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:02.134041+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16277504,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10699","reported_seq":10793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:04.937556+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:04.937556+0000","last_peered":"2024-09-11T05:05:04.937556+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:04.937556+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16635040,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-276}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10829","reported_seq":15687,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:03.207127+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:03.207127+0000","last_peered":"2024-09-11T05:05:03.207127+0000","last_clean":"2024-09-11T05:05:03.207127+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:03.207127+0000","last_undegraded":"2024-09-11T05:05:03.207127+0000","last_fullsized":"2024-09-11T05:05:03.207127+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16334848,"num_objects":1994,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1539,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10745","reported_seq":10867,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.258848+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:02.258848+0000","last_peered":"2024-09-11T05:05:02.258848+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:02.258848+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16730432,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10553","reported_seq":10665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.741676+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:02.741676+0000","last_peered":"2024-09-11T05:05:02.741676+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:02.741676+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16176288,"num_objects":1975,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10551,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10658","reported_seq":10775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.247599+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:02.247599+0000","last_peered":"2024-09-11T05:05:02.247599+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:02.247599+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338672,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10655,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10368","reported_seq":14969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:02.837089+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:02.837089+0000","last_peered":"2024-09-11T05:05:02.837089+0000","last_clean":"2024-09-11T05:05:02.837089+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:02.837089+0000","last_undegraded":"2024-09-11T05:05:02.837089+0000","last_fullsized":"2024-09-11T05:05:02.837089+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":2016,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10716","reported_seq":13281,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.812762+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:02.812762+0000","last_peered":"2024-09-11T05:05:02.812762+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:02.812762+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16439888,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10715,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":914}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10609","reported_seq":10716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:01.962299+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:01.962299+0000","last_peered":"2024-09-11T05:05:01.962299+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:01.962299+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16241824,"num_objects":1983,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10607,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10525","reported_seq":10644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.720821+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:02.720821+0000","last_peered":"2024-09-11T05:05:02.720821+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:02.720821+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10772","reported_seq":10891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.046913+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:05.046913+0000","last_peered":"2024-09-11T05:05:05.046913+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:05.046913+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16422048,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10770,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":261149160,"num_objects":31884,"num_object_clones":0,"num_object_copies":127536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154019,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":521949184,"data_stored":521949184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764740,"kb_used_data":129140,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93607100,"statfs":{"total":96636764160,"available":95853670400,"internally_reserved":0,"allocated":132239360,"data_stored":132050158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,0,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745568,"kb_used_data":128496,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93626272,"statfs":{"total":96636764160,"available":95873302528,"internally_reserved":0,"allocated":131579904,"data_stored":131390702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589224,"kb_used_data":128600,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93782616,"statfs":{"total":96636764160,"available":96033398784,"internally_reserved":0,"allocated":131686400,"data_stored":131497198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471658194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740116,"kb_used_data":126780,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93631724,"statfs":{"total":96636764160,"available":95878885376,"internally_reserved":0,"allocated":129822720,"data_stored":129633518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":129032192,"data_stored":129032192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130654208,"data_stored":130654208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130928640,"data_stored":130928640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":131334144,"data_stored":131334144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:05.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:05.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10477", 'reported_seq': 10572, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.279695+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:02.279695+0000', 'last_peered': '2024-09-11T05:05:02.279695+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:02.279695+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16687104, 'num_objects': 2037, 'num_object_clones': 0, 'num_object_copies': 8148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10477, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -254}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:05.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:05.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10702", 'reported_seq': 10813, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:05.144031+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:05.144031+0000', 'last_peered': '2024-09-11T05:05:05.144031+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:05.144031+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16676000, 'num_objects': 2036, 'num_object_clones': 0, 'num_object_copies': 8144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10700, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -222}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:05.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:05.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10179", 'reported_seq': 10284, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.650335+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:02.650335+0000', 'last_peered': '2024-09-11T05:05:02.650335+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:02.650335+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16226920, 'num_objects': 1982, 'num_object_clones': 0, 'num_object_copies': 7928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1982, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10178, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -229}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:05.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:05.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10495", 'reported_seq': 10603, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:05.160527+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:05.160527+0000', 'last_peered': '2024-09-11T05:05:05.160527+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:05.160527+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15733920, 'num_objects': 1921, 'num_object_clones': 0, 'num_object_copies': 7684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10493, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10543", 'reported_seq': 10666, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.303956+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:02.303956+0000', 'last_peered': '2024-09-11T05:05:02.303956+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:02.303956+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16282784, 'num_objects': 1988, 'num_object_clones': 0, 'num_object_copies': 7952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10541, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -233}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10653", 'reported_seq': 10759, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.134041+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:02.134041+0000', 'last_peered': '2024-09-11T05:05:02.134041+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:02.134041+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16277504, 'num_objects': 1987, 'num_object_clones': 0, 'num_object_copies': 7948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10653, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -307}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10699", 'reported_seq': 10793, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:04.937556+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:04.937556+0000', 'last_peered': '2024-09-11T05:05:04.937556+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:04.937556+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16635040, 'num_objects': 2031, 'num_object_clones': 0, 'num_object_copies': 8124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10697, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -276}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10745", 'reported_seq': 10867, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.258848+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:02.258848+0000', 'last_peered': '2024-09-11T05:05:02.258848+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:02.258848+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16730432, 'num_objects': 2043, 'num_object_clones': 0, 'num_object_copies': 8172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10742, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -323}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10553", 'reported_seq': 10665, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.741676+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:02.741676+0000', 'last_peered': '2024-09-11T05:05:02.741676+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:02.741676+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16176288, 'num_objects': 1975, 'num_object_clones': 0, 'num_object_copies': 7900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10551, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -331}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:05.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10658", 'reported_seq': 10775, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.247599+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:02.247599+0000', 'last_peered': '2024-09-11T05:05:02.247599+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:02.247599+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16338672, 'num_objects': 1995, 'num_object_clones': 0, 'num_object_copies': 7980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10655, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -303}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10716", 'reported_seq': 13281, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.812762+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:02.812762+0000', 'last_peered': '2024-09-11T05:05:02.812762+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:02.812762+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16439888, 'num_objects': 2007, 'num_object_clones': 0, 'num_object_copies': 8028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1093, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10715, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1241, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 914}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10609", 'reported_seq': 10716, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:01.962299+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:01.962299+0000', 'last_peered': '2024-09-11T05:05:01.962299+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:01.962299+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16241824, 'num_objects': 1983, 'num_object_clones': 0, 'num_object_copies': 7932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10607, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10525", 'reported_seq': 10644, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:02.720821+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:02.720821+0000', 'last_peered': '2024-09-11T05:05:02.720821+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:02.720821+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15430816, 'num_objects': 1884, 'num_object_clones': 0, 'num_object_copies': 7536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10523, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:05.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10772", 'reported_seq': 10891, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:05.046913+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:05.046913+0000', 'last_peered': '2024-09-11T05:05:05.046913+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:05.046913+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16422048, 'num_objects': 2005, 'num_object_clones': 0, 'num_object_copies': 8020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10770, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:05.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:06.698 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30784/127536 objects degraded (24.137%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":633,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":31884,"data_bytes":261149160,"bytes_used":2907799552,"bytes_avail":383639257088,"bytes_total":386547056640,"degraded_objects":30784,"degraded_total":127536,"degraded_ratio":0.2413749843181533,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":183},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:06.716 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:06.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:07.334 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:07.334 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:07.361 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-11T05:05:05.853550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":261149160,"num_objects":31884,"num_object_clones":0,"num_object_copies":127536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154019,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839648,"kb_used_data":513016,"kb_used_omap":14,"kb_used_meta":2326513,"kb_avail":374647712,"statfs":{"total":386547056640,"available":383639257088,"internally_reserved":0,"allocated":525328384,"data_stored":524571576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2382349878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,0,16,16,14,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18087936,"num_objects":-2208,"num_object_clones":0,"num_object_copies":-8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004282"},"pg_stats":[{"pgid":"1.f","version":"35'10477","reported_seq":10572,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.279695+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:02.279695+0000","last_peered":"2024-09-11T05:05:02.279695+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:02.279695+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16687104,"num_objects":2037,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10702","reported_seq":10813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.144031+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:05.144031+0000","last_peered":"2024-09-11T05:05:05.144031+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:05.144031+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16676000,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10179","reported_seq":10284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.650335+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:02.650335+0000","last_peered":"2024-09-11T05:05:02.650335+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:02.650335+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16226920,"num_objects":1982,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10178,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-229}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10495","reported_seq":10603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.160527+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:05.160527+0000","last_peered":"2024-09-11T05:05:05.160527+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:05.160527+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15733920,"num_objects":1921,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10543","reported_seq":10666,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.303956+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:02.303956+0000","last_peered":"2024-09-11T05:05:02.303956+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:02.303956+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282784,"num_objects":1988,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10653","reported_seq":10759,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.134041+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:02.134041+0000","last_peered":"2024-09-11T05:05:02.134041+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:02.134041+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16277504,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10699","reported_seq":10793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:04.937556+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:04.937556+0000","last_peered":"2024-09-11T05:05:04.937556+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:04.937556+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16635040,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-276}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10829","reported_seq":15687,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:03.207127+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:03.207127+0000","last_peered":"2024-09-11T05:05:03.207127+0000","last_clean":"2024-09-11T05:05:03.207127+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:03.207127+0000","last_undegraded":"2024-09-11T05:05:03.207127+0000","last_fullsized":"2024-09-11T05:05:03.207127+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16334848,"num_objects":1994,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1539,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10745","reported_seq":10867,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.258848+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:02.258848+0000","last_peered":"2024-09-11T05:05:02.258848+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:02.258848+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16730432,"num_objects":2043,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10553","reported_seq":10665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.741676+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:02.741676+0000","last_peered":"2024-09-11T05:05:02.741676+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:02.741676+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16176288,"num_objects":1975,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10551,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10658","reported_seq":10775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.247599+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:02.247599+0000","last_peered":"2024-09-11T05:05:02.247599+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:02.247599+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16338672,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10655,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10368","reported_seq":14969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:02.837089+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:02.837089+0000","last_peered":"2024-09-11T05:05:02.837089+0000","last_clean":"2024-09-11T05:05:02.837089+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:02.837089+0000","last_undegraded":"2024-09-11T05:05:02.837089+0000","last_fullsized":"2024-09-11T05:05:02.837089+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":2016,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10716","reported_seq":13281,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.812762+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:02.812762+0000","last_peered":"2024-09-11T05:05:02.812762+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:02.812762+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16439888,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10715,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":914}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10609","reported_seq":10716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:01.962299+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:01.962299+0000","last_peered":"2024-09-11T05:05:01.962299+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:01.962299+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16241824,"num_objects":1983,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10607,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10525","reported_seq":10644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:02.720821+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:02.720821+0000","last_peered":"2024-09-11T05:05:02.720821+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:02.720821+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430816,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10772","reported_seq":10891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.046913+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:05.046913+0000","last_peered":"2024-09-11T05:05:05.046913+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:05.046913+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16422048,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10770,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":261149160,"num_objects":31884,"num_object_clones":0,"num_object_copies":127536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154019,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":521949184,"data_stored":521949184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764740,"kb_used_data":129140,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93607100,"statfs":{"total":96636764160,"available":95853670400,"internally_reserved":0,"allocated":132239360,"data_stored":132050158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,0,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745568,"kb_used_data":128496,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93626272,"statfs":{"total":96636764160,"available":95873302528,"internally_reserved":0,"allocated":131579904,"data_stored":131390702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589224,"kb_used_data":128600,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93782616,"statfs":{"total":96636764160,"available":96033398784,"internally_reserved":0,"allocated":131686400,"data_stored":131497198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":471658194},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740116,"kb_used_data":126780,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93631724,"statfs":{"total":96636764160,"available":95878885376,"internally_reserved":0,"allocated":129822720,"data_stored":129633518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":129032192,"data_stored":129032192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130654208,"data_stored":130654208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130928640,"data_stored":130928640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":131334144,"data_stored":131334144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:10.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:10.986 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:11.006 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-11T05:05:09.855157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":255128040,"num_objects":31149,"num_object_clones":0,"num_object_copies":124596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31149,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154754,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819792,"kb_used_data":501476,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374667568,"statfs":{"total":386547056640,"available":383659589632,"internally_reserved":0,"allocated":513511424,"data_stored":512754616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2373830198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,9,7,5,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17170432,"num_objects":-2096,"num_object_clones":0,"num_object_copies":-8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2096,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004159"},"pg_stats":[{"pgid":"1.f","version":"35'10538","reported_seq":10633,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.444101+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:07.444101+0000","last_peered":"2024-09-11T05:05:07.444101+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:07.444101+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10538,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10702","reported_seq":10813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.144031+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:05.144031+0000","last_peered":"2024-09-11T05:05:05.144031+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:05.144031+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16676000,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10242","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.782894+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:07.782894+0000","last_peered":"2024-09-11T05:05:07.782894+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:07.782894+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15710824,"num_objects":1919,"num_object_clones":0,"num_object_copies":7676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10241,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-292}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10495","reported_seq":10603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.160527+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:05.160527+0000","last_peered":"2024-09-11T05:05:05.160527+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:05.160527+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15733920,"num_objects":1921,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10602","reported_seq":10725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.491632+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:07.491632+0000","last_peered":"2024-09-11T05:05:07.491632+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:07.491632+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15799456,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-292}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10713","reported_seq":10818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.442595+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:07.442595+0000","last_peered":"2024-09-11T05:05:07.442595+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:07.442595+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15785984,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10713,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10699","reported_seq":10793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:04.937556+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:04.937556+0000","last_peered":"2024-09-11T05:05:04.937556+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:04.937556+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16635040,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-276}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10885","reported_seq":15742,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:08.240032+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:08.240032+0000","last_peered":"2024-09-11T05:05:08.240032+0000","last_clean":"2024-09-11T05:05:08.240032+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:08.240032+0000","last_undegraded":"2024-09-11T05:05:08.240032+0000","last_fullsized":"2024-09-11T05:05:08.240032+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15876096,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10810","reported_seq":10933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.296708+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:07.296708+0000","last_peered":"2024-09-11T05:05:07.296708+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:07.296708+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16197952,"num_objects":1978,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-388}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10603","reported_seq":10715,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.655331+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:07.655331+0000","last_peered":"2024-09-11T05:05:07.655331+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:07.655331+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15766688,"num_objects":1925,"num_object_clones":0,"num_object_copies":7700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-381}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10723","reported_seq":10840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.397517+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:07.397517+0000","last_peered":"2024-09-11T05:05:07.397517+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:07.397517+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806192,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10720,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-368}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10431","reported_seq":15032,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:08.079088+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:08.079088+0000","last_peered":"2024-09-11T05:05:08.079088+0000","last_clean":"2024-09-11T05:05:08.079088+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:08.079088+0000","last_undegraded":"2024-09-11T05:05:08.079088+0000","last_fullsized":"2024-09-11T05:05:08.079088+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15998976,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10795","reported_seq":13463,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.818356+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:07.818356+0000","last_peered":"2024-09-11T05:05:07.818356+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:07.818356+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15792720,"num_objects":1928,"num_object_clones":0,"num_object_copies":7712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10794,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10671","reported_seq":10778,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.408555+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:07.408555+0000","last_peered":"2024-09-11T05:05:07.408555+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:07.408555+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15733920,"num_objects":1921,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-343}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10577","reported_seq":10693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.894508+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:07.894508+0000","last_peered":"2024-09-11T05:05:07.894508+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:07.894508+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15004832,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10575,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10772","reported_seq":10891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.046913+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:05.046913+0000","last_peered":"2024-09-11T05:05:05.046913+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:05.046913+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16422048,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10770,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":255128040,"num_objects":31149,"num_object_clones":0,"num_object_copies":124596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31149,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154754,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":510119936,"data_stored":510119936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748408,"kb_used_data":125288,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93623432,"statfs":{"total":96636764160,"available":95870394368,"internally_reserved":0,"allocated":128294912,"data_stored":128105710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741764,"kb_used_data":124696,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93630076,"statfs":{"total":96636764160,"available":95877197824,"internally_reserved":0,"allocated":127688704,"data_stored":127499502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589504,"kb_used_data":124712,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93782336,"statfs":{"total":96636764160,"available":96033112064,"internally_reserved":0,"allocated":127705088,"data_stored":127515886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475918034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740116,"kb_used_data":126780,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93631724,"statfs":{"total":96636764160,"available":95878885376,"internally_reserved":0,"allocated":129822720,"data_stored":129633518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":129032192,"data_stored":129032192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126730240,"data_stored":126730240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":127037440,"data_stored":127037440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127320064,"data_stored":127320064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:11.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:11.681 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:11.703 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-11T05:05:09.855157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":255128040,"num_objects":31149,"num_object_clones":0,"num_object_copies":124596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31149,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154754,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819792,"kb_used_data":501476,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374667568,"statfs":{"total":386547056640,"available":383659589632,"internally_reserved":0,"allocated":513511424,"data_stored":512754616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2373830198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,9,7,5,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17170432,"num_objects":-2096,"num_object_clones":0,"num_object_copies":-8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2096,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004159"},"pg_stats":[{"pgid":"1.f","version":"35'10538","reported_seq":10633,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.444101+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:07.444101+0000","last_peered":"2024-09-11T05:05:07.444101+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:07.444101+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10538,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10702","reported_seq":10813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.144031+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:05.144031+0000","last_peered":"2024-09-11T05:05:05.144031+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:05.144031+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16676000,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10242","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.782894+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:07.782894+0000","last_peered":"2024-09-11T05:05:07.782894+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:07.782894+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15710824,"num_objects":1919,"num_object_clones":0,"num_object_copies":7676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10241,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-292}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10495","reported_seq":10603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.160527+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:05.160527+0000","last_peered":"2024-09-11T05:05:05.160527+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:05.160527+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15733920,"num_objects":1921,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10602","reported_seq":10725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.491632+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:07.491632+0000","last_peered":"2024-09-11T05:05:07.491632+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:07.491632+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15799456,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-292}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10713","reported_seq":10818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.442595+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:07.442595+0000","last_peered":"2024-09-11T05:05:07.442595+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:07.442595+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15785984,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10713,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10699","reported_seq":10793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:04.937556+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:04.937556+0000","last_peered":"2024-09-11T05:05:04.937556+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:04.937556+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16635040,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-276}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10885","reported_seq":15742,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:08.240032+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:08.240032+0000","last_peered":"2024-09-11T05:05:08.240032+0000","last_clean":"2024-09-11T05:05:08.240032+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:08.240032+0000","last_undegraded":"2024-09-11T05:05:08.240032+0000","last_fullsized":"2024-09-11T05:05:08.240032+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15876096,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10810","reported_seq":10933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.296708+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:07.296708+0000","last_peered":"2024-09-11T05:05:07.296708+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:07.296708+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16197952,"num_objects":1978,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-388}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10603","reported_seq":10715,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.655331+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:07.655331+0000","last_peered":"2024-09-11T05:05:07.655331+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:07.655331+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15766688,"num_objects":1925,"num_object_clones":0,"num_object_copies":7700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-381}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10723","reported_seq":10840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.397517+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:07.397517+0000","last_peered":"2024-09-11T05:05:07.397517+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:07.397517+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806192,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10720,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-368}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10431","reported_seq":15032,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:08.079088+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:08.079088+0000","last_peered":"2024-09-11T05:05:08.079088+0000","last_clean":"2024-09-11T05:05:08.079088+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:08.079088+0000","last_undegraded":"2024-09-11T05:05:08.079088+0000","last_fullsized":"2024-09-11T05:05:08.079088+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15998976,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10795","reported_seq":13463,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.818356+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:07.818356+0000","last_peered":"2024-09-11T05:05:07.818356+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:07.818356+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15792720,"num_objects":1928,"num_object_clones":0,"num_object_copies":7712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10794,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10671","reported_seq":10778,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.408555+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:07.408555+0000","last_peered":"2024-09-11T05:05:07.408555+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:07.408555+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15733920,"num_objects":1921,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-343}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10577","reported_seq":10693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.894508+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:07.894508+0000","last_peered":"2024-09-11T05:05:07.894508+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:07.894508+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15004832,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10575,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10772","reported_seq":10891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:05.046913+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:05.046913+0000","last_peered":"2024-09-11T05:05:05.046913+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:05.046913+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16422048,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10770,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":255128040,"num_objects":31149,"num_object_clones":0,"num_object_copies":124596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31149,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154754,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":510119936,"data_stored":510119936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748408,"kb_used_data":125288,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93623432,"statfs":{"total":96636764160,"available":95870394368,"internally_reserved":0,"allocated":128294912,"data_stored":128105710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741764,"kb_used_data":124696,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93630076,"statfs":{"total":96636764160,"available":95877197824,"internally_reserved":0,"allocated":127688704,"data_stored":127499502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589504,"kb_used_data":124712,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93782336,"statfs":{"total":96636764160,"available":96033112064,"internally_reserved":0,"allocated":127705088,"data_stored":127515886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475918034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740116,"kb_used_data":126780,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93631724,"statfs":{"total":96636764160,"available":95878885376,"internally_reserved":0,"allocated":129822720,"data_stored":129633518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":129032192,"data_stored":129032192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126730240,"data_stored":126730240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":127037440,"data_stored":127037440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127320064,"data_stored":127320064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:11.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:11.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10538", 'reported_seq': 10633, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.444101+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:07.444101+0000', 'last_peered': '2024-09-11T05:05:07.444101+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:07.444101+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16187392, 'num_objects': 1976, 'num_object_clones': 0, 'num_object_copies': 7904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10538, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:11.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:11.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10702", 'reported_seq': 10813, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:05.144031+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:05.144031+0000', 'last_peered': '2024-09-11T05:05:05.144031+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:05.144031+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16676000, 'num_objects': 2036, 'num_object_clones': 0, 'num_object_copies': 8144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10700, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -222}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:11.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:11.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10242", 'reported_seq': 10347, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.782894+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:07.782894+0000', 'last_peered': '2024-09-11T05:05:07.782894+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:07.782894+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15710824, 'num_objects': 1919, 'num_object_clones': 0, 'num_object_copies': 7676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1919, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10241, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -292}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10495", 'reported_seq': 10603, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:05.160527+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:05.160527+0000', 'last_peered': '2024-09-11T05:05:05.160527+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:05.160527+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15733920, 'num_objects': 1921, 'num_object_clones': 0, 'num_object_copies': 7684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10493, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10602", 'reported_seq': 10725, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.491632+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:07.491632+0000', 'last_peered': '2024-09-11T05:05:07.491632+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:07.491632+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15799456, 'num_objects': 1929, 'num_object_clones': 0, 'num_object_copies': 7716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10600, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -292}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10713", 'reported_seq': 10818, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.442595+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:07.442595+0000', 'last_peered': '2024-09-11T05:05:07.442595+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:07.442595+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15785984, 'num_objects': 1927, 'num_object_clones': 0, 'num_object_copies': 7708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10713, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10699", 'reported_seq': 10793, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:04.937556+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:04.937556+0000', 'last_peered': '2024-09-11T05:05:04.937556+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:04.937556+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16635040, 'num_objects': 2031, 'num_object_clones': 0, 'num_object_copies': 8124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10697, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -276}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10810", 'reported_seq': 10933, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.296708+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:07.296708+0000', 'last_peered': '2024-09-11T05:05:07.296708+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:07.296708+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16197952, 'num_objects': 1978, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10807, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -388}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:11.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10603", 'reported_seq': 10715, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.655331+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:07.655331+0000', 'last_peered': '2024-09-11T05:05:07.655331+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:07.655331+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15766688, 'num_objects': 1925, 'num_object_clones': 0, 'num_object_copies': 7700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10601, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -381}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10723", 'reported_seq': 10840, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.397517+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:07.397517+0000', 'last_peered': '2024-09-11T05:05:07.397517+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:07.397517+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15806192, 'num_objects': 1930, 'num_object_clones': 0, 'num_object_copies': 7720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10720, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -368}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10795", 'reported_seq': 13463, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.818356+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:07.818356+0000', 'last_peered': '2024-09-11T05:05:07.818356+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:07.818356+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15792720, 'num_objects': 1928, 'num_object_clones': 0, 'num_object_copies': 7712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1042, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10794, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1292, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10671", 'reported_seq': 10778, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.408555+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:07.408555+0000', 'last_peered': '2024-09-11T05:05:07.408555+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:07.408555+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15733920, 'num_objects': 1921, 'num_object_clones': 0, 'num_object_copies': 7684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10669, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10577", 'reported_seq': 10693, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:07.894508+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:07.894508+0000', 'last_peered': '2024-09-11T05:05:07.894508+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:07.894508+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15004832, 'num_objects': 1832, 'num_object_clones': 0, 'num_object_copies': 7328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10575, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:11.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:11.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10772", 'reported_seq': 10891, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:05.046913+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:05.046913+0000', 'last_peered': '2024-09-11T05:05:05.046913+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:05.046913+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16422048, 'num_objects': 2005, 'num_object_clones': 0, 'num_object_copies': 8020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10770, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:11.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:12.561 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:12.561 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30733/123624 objects degraded (24.860%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":639,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30906,"data_bytes":253137384,"bytes_used":2887692288,"bytes_avail":383659364352,"bytes_total":386547056640,"degraded_objects":30733,"degraded_total":123624,"degraded_ratio":0.24860059535365303,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":184},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:12.585 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:12.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:13.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:13.203 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:13.226 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-11T05:05:11.856055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253137384,"num_objects":30906,"num_object_clones":0,"num_object_copies":123624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154997,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2820012,"kb_used_data":497544,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374667348,"statfs":{"total":386547056640,"available":383659364352,"internally_reserved":0,"allocated":509485056,"data_stored":508728248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2378090038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,9,10,7,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18145280,"num_objects":-2215,"num_object_clones":0,"num_object_copies":-8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2215,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004371"},"pg_stats":[{"pgid":"1.f","version":"35'10538","reported_seq":10633,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.444101+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:07.444101+0000","last_peered":"2024-09-11T05:05:07.444101+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:07.444101+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":1976,"num_object_clones":0,"num_object_copies":7904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10538,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10769","reported_seq":10882,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:09.984830+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:09.984830+0000","last_peered":"2024-09-11T05:05:09.984830+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:09.984830+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16127136,"num_objects":1969,"num_object_clones":0,"num_object_copies":7876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10242","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.782894+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:07.782894+0000","last_peered":"2024-09-11T05:05:07.782894+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:07.782894+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15710824,"num_objects":1919,"num_object_clones":0,"num_object_copies":7676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10241,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-292}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10548","reported_seq":10656,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:10.101459+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:10.101459+0000","last_peered":"2024-09-11T05:05:10.101459+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:10.101459+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15299744,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10602","reported_seq":10725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.491632+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:07.491632+0000","last_peered":"2024-09-11T05:05:07.491632+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:07.491632+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15799456,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-292}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10713","reported_seq":10818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.442595+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:07.442595+0000","last_peered":"2024-09-11T05:05:07.442595+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:07.442595+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15785984,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10713,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10755","reported_seq":10849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:10.089992+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:10.089992+0000","last_peered":"2024-09-11T05:05:10.089992+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:10.089992+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16176288,"num_objects":1975,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10753,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-332}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10885","reported_seq":15742,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:08.240032+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:08.240032+0000","last_peered":"2024-09-11T05:05:08.240032+0000","last_clean":"2024-09-11T05:05:08.240032+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:08.240032+0000","last_undegraded":"2024-09-11T05:05:08.240032+0000","last_fullsized":"2024-09-11T05:05:08.240032+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15876096,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10810","reported_seq":10933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.296708+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:07.296708+0000","last_peered":"2024-09-11T05:05:07.296708+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:07.296708+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16197952,"num_objects":1978,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-388}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10603","reported_seq":10715,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.655331+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:07.655331+0000","last_peered":"2024-09-11T05:05:07.655331+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:07.655331+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15766688,"num_objects":1925,"num_object_clones":0,"num_object_copies":7700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-381}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10723","reported_seq":10840,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.397517+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:07.397517+0000","last_peered":"2024-09-11T05:05:07.397517+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:07.397517+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806192,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10720,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-368}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10431","reported_seq":15032,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:08.079088+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:08.079088+0000","last_peered":"2024-09-11T05:05:08.079088+0000","last_clean":"2024-09-11T05:05:08.079088+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:08.079088+0000","last_undegraded":"2024-09-11T05:05:08.079088+0000","last_fullsized":"2024-09-11T05:05:08.079088+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15998976,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10795","reported_seq":13463,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.818356+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:07.818356+0000","last_peered":"2024-09-11T05:05:07.818356+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:07.818356+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15792720,"num_objects":1928,"num_object_clones":0,"num_object_copies":7712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10794,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10671","reported_seq":10778,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.408555+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:07.408555+0000","last_peered":"2024-09-11T05:05:07.408555+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:07.408555+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15733920,"num_objects":1921,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-343}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10577","reported_seq":10693,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:07.894508+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:07.894508+0000","last_peered":"2024-09-11T05:05:07.894508+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:07.894508+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15004832,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10575,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10839","reported_seq":10957,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:10.121991+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:10.121991+0000","last_peered":"2024-09-11T05:05:10.121991+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:10.121991+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15873184,"num_objects":1938,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10837,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":253137384,"num_objects":30906,"num_object_clones":0,"num_object_copies":123624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154997,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":506171392,"data_stored":506171392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748408,"kb_used_data":125288,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93623432,"statfs":{"total":96636764160,"available":95870394368,"internally_reserved":0,"allocated":128294912,"data_stored":128105710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741764,"kb_used_data":124696,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93630076,"statfs":{"total":96636764160,"available":95877197824,"internally_reserved":0,"allocated":127688704,"data_stored":127499502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589504,"kb_used_data":124712,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93782336,"statfs":{"total":96636764160,"available":96033112064,"internally_reserved":0,"allocated":127705088,"data_stored":127515886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":475918034},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740336,"kb_used_data":122848,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93631504,"statfs":{"total":96636764160,"available":95878660096,"internally_reserved":0,"allocated":125796352,"data_stored":125607150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125083648,"data_stored":125083648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126730240,"data_stored":126730240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":127037440,"data_stored":127037440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127320064,"data_stored":127320064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:16.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:16.816 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:16.836 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-11T05:05:15.857700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":245338600,"num_objects":29954,"num_object_clones":0,"num_object_copies":119816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155949,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2787084,"kb_used_data":481292,"kb_used_omap":14,"kb_used_meta":2305713,"kb_avail":374700276,"statfs":{"total":386547056640,"available":383693082624,"internally_reserved":0,"allocated":492843008,"data_stored":492086200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2361050678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,7,11,16,10,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17997824,"num_objects":-2197,"num_object_clones":0,"num_object_copies":-8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004671"},"pg_stats":[{"pgid":"1.f","version":"35'10607","reported_seq":10702,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.653815+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:12.653815+0000","last_peered":"2024-09-11T05:05:12.653815+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:12.653815+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15622144,"num_objects":1907,"num_object_clones":0,"num_object_copies":7628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10607,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10834","reported_seq":10949,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.052365+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:15.052365+0000","last_peered":"2024-09-11T05:05:15.052365+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:15.052365+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15594656,"num_objects":1904,"num_object_clones":0,"num_object_copies":7616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-354}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10302","reported_seq":10407,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.775293+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:12.775293+0000","last_peered":"2024-09-11T05:05:12.775293+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:12.775293+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15219304,"num_objects":1859,"num_object_clones":0,"num_object_copies":7436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10301,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10607","reported_seq":10714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.185818+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:15.185818+0000","last_peered":"2024-09-11T05:05:15.185818+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:15.185818+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816416,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-452}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10659","reported_seq":10782,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.393474+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:12.393474+0000","last_peered":"2024-09-11T05:05:12.393474+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:12.393474+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15332512,"num_objects":1872,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10782","reported_seq":10887,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.687400+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:12.687400+0000","last_peered":"2024-09-11T05:05:12.687400+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:12.687400+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15220736,"num_objects":1858,"num_object_clones":0,"num_object_copies":7432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10782,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10797","reported_seq":10890,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:14.935482+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:14.935482+0000","last_peered":"2024-09-11T05:05:14.935482+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:14.935482+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15832224,"num_objects":1933,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10795,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-374}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10943","reported_seq":15801,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:13.084588+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:13.084588+0000","last_peered":"2024-09-11T05:05:13.084588+0000","last_clean":"2024-09-11T05:05:13.084588+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:13.084588+0000","last_undegraded":"2024-09-11T05:05:13.084588+0000","last_fullsized":"2024-09-11T05:05:13.084588+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1653,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10856","reported_seq":10979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.575314+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:12.575314+0000","last_peered":"2024-09-11T05:05:12.575314+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:12.575314+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15821120,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10853,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10667","reported_seq":10779,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.865089+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:12.865089+0000","last_peered":"2024-09-11T05:05:12.865089+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:12.865089+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15242400,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-445}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10780","reported_seq":10897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.560107+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:12.560107+0000","last_peered":"2024-09-11T05:05:12.560107+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:12.560107+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15339248,"num_objects":1873,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10499","reported_seq":15100,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:13.134241+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:13.134241+0000","last_peered":"2024-09-11T05:05:13.134241+0000","last_clean":"2024-09-11T05:05:13.134241+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:13.134241+0000","last_undegraded":"2024-09-11T05:05:13.134241+0000","last_fullsized":"2024-09-11T05:05:13.134241+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15441920,"num_objects":1885,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10853","reported_seq":13641,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.926963+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:12.926963+0000","last_peered":"2024-09-11T05:05:12.926963+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:12.926963+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15317584,"num_objects":1870,"num_object_clones":0,"num_object_copies":7480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":888}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10726","reported_seq":10833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.516265+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:12.516265+0000","last_peered":"2024-09-11T05:05:12.516265+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:12.516265+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15283360,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-398}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10640","reported_seq":10759,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.797917+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:12.797917+0000","last_peered":"2024-09-11T05:05:12.797917+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:12.797917+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488736,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-534}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10901","reported_seq":11021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.064299+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:15.064299+0000","last_peered":"2024-09-11T05:05:15.064299+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:15.064299+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15365280,"num_objects":1876,"num_object_clones":0,"num_object_copies":7504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10899,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":245338600,"num_objects":29954,"num_object_clones":0,"num_object_copies":119816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155949,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":490168320,"data_stored":490168320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747992,"kb_used_data":120712,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93623848,"statfs":{"total":96636764160,"available":95870820352,"internally_reserved":0,"allocated":123609088,"data_stored":123419886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725328,"kb_used_data":120768,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93646512,"statfs":{"total":96636764160,"available":95894028288,"internally_reserved":0,"allocated":123666432,"data_stored":123477230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589748,"kb_used_data":120804,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93782092,"statfs":{"total":96636764160,"available":96032862208,"internally_reserved":0,"allocated":123703296,"data_stored":123514094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480177874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724016,"kb_used_data":119008,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93647824,"statfs":{"total":96636764160,"available":95895371776,"internally_reserved":0,"allocated":121864192,"data_stored":121674990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619507407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121135104,"data_stored":121135104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122822656,"data_stored":122822656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123006976,"data_stored":123006976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":123203584,"data_stored":123203584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:16.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:17.492 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:17.492 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:17.519 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-11T05:05:15.857700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":245338600,"num_objects":29954,"num_object_clones":0,"num_object_copies":119816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155949,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2787084,"kb_used_data":481292,"kb_used_omap":14,"kb_used_meta":2305713,"kb_avail":374700276,"statfs":{"total":386547056640,"available":383693082624,"internally_reserved":0,"allocated":492843008,"data_stored":492086200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2361050678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,7,11,16,10,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17997824,"num_objects":-2197,"num_object_clones":0,"num_object_copies":-8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004671"},"pg_stats":[{"pgid":"1.f","version":"35'10607","reported_seq":10702,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.653815+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:12.653815+0000","last_peered":"2024-09-11T05:05:12.653815+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:12.653815+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15622144,"num_objects":1907,"num_object_clones":0,"num_object_copies":7628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10607,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10834","reported_seq":10949,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.052365+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:15.052365+0000","last_peered":"2024-09-11T05:05:15.052365+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:15.052365+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15594656,"num_objects":1904,"num_object_clones":0,"num_object_copies":7616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-354}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10302","reported_seq":10407,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.775293+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:12.775293+0000","last_peered":"2024-09-11T05:05:12.775293+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:12.775293+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15219304,"num_objects":1859,"num_object_clones":0,"num_object_copies":7436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10301,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10607","reported_seq":10714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.185818+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:15.185818+0000","last_peered":"2024-09-11T05:05:15.185818+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:15.185818+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816416,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-452}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10659","reported_seq":10782,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.393474+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:12.393474+0000","last_peered":"2024-09-11T05:05:12.393474+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:12.393474+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15332512,"num_objects":1872,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10782","reported_seq":10887,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.687400+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:12.687400+0000","last_peered":"2024-09-11T05:05:12.687400+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:12.687400+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15220736,"num_objects":1858,"num_object_clones":0,"num_object_copies":7432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10782,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-436}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10797","reported_seq":10890,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:14.935482+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:14.935482+0000","last_peered":"2024-09-11T05:05:14.935482+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:14.935482+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15832224,"num_objects":1933,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10795,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-374}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10943","reported_seq":15801,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:13.084588+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:13.084588+0000","last_peered":"2024-09-11T05:05:13.084588+0000","last_clean":"2024-09-11T05:05:13.084588+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:13.084588+0000","last_undegraded":"2024-09-11T05:05:13.084588+0000","last_fullsized":"2024-09-11T05:05:13.084588+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1653,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10856","reported_seq":10979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.575314+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:12.575314+0000","last_peered":"2024-09-11T05:05:12.575314+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:12.575314+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15821120,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10853,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10667","reported_seq":10779,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.865089+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:12.865089+0000","last_peered":"2024-09-11T05:05:12.865089+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:12.865089+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15242400,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-445}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10780","reported_seq":10897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.560107+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:12.560107+0000","last_peered":"2024-09-11T05:05:12.560107+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:12.560107+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15339248,"num_objects":1873,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10499","reported_seq":15100,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:13.134241+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:13.134241+0000","last_peered":"2024-09-11T05:05:13.134241+0000","last_clean":"2024-09-11T05:05:13.134241+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:13.134241+0000","last_undegraded":"2024-09-11T05:05:13.134241+0000","last_fullsized":"2024-09-11T05:05:13.134241+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15441920,"num_objects":1885,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10853","reported_seq":13641,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.926963+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:12.926963+0000","last_peered":"2024-09-11T05:05:12.926963+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:12.926963+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15317584,"num_objects":1870,"num_object_clones":0,"num_object_copies":7480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":888}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10726","reported_seq":10833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.516265+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:12.516265+0000","last_peered":"2024-09-11T05:05:12.516265+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:12.516265+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15283360,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-398}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10640","reported_seq":10759,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.797917+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:12.797917+0000","last_peered":"2024-09-11T05:05:12.797917+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:12.797917+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488736,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-534}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10901","reported_seq":11021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.064299+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:15.064299+0000","last_peered":"2024-09-11T05:05:15.064299+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:15.064299+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15365280,"num_objects":1876,"num_object_clones":0,"num_object_copies":7504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10899,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":245338600,"num_objects":29954,"num_object_clones":0,"num_object_copies":119816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155949,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":490168320,"data_stored":490168320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747992,"kb_used_data":120712,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93623848,"statfs":{"total":96636764160,"available":95870820352,"internally_reserved":0,"allocated":123609088,"data_stored":123419886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725328,"kb_used_data":120768,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93646512,"statfs":{"total":96636764160,"available":95894028288,"internally_reserved":0,"allocated":123666432,"data_stored":123477230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589748,"kb_used_data":120804,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93782092,"statfs":{"total":96636764160,"available":96032862208,"internally_reserved":0,"allocated":123703296,"data_stored":123514094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480177874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724016,"kb_used_data":119008,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93647824,"statfs":{"total":96636764160,"available":95895371776,"internally_reserved":0,"allocated":121864192,"data_stored":121674990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619507407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121135104,"data_stored":121135104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122822656,"data_stored":122822656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123006976,"data_stored":123006976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":123203584,"data_stored":123203584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:17.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:17.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10607", 'reported_seq': 10702, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.653815+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:12.653815+0000', 'last_peered': '2024-09-11T05:05:12.653815+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:12.653815+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15622144, 'num_objects': 1907, 'num_object_clones': 0, 'num_object_copies': 7628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10607, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -384}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:17.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:17.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10834", 'reported_seq': 10949, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:15.052365+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:15.052365+0000', 'last_peered': '2024-09-11T05:05:15.052365+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:15.052365+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15594656, 'num_objects': 1904, 'num_object_clones': 0, 'num_object_copies': 7616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10832, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -354}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10302", 'reported_seq': 10407, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.775293+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:12.775293+0000', 'last_peered': '2024-09-11T05:05:12.775293+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:12.775293+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15219304, 'num_objects': 1859, 'num_object_clones': 0, 'num_object_copies': 7436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1859, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10301, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -352}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10607", 'reported_seq': 10714, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:15.185818+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:15.185818+0000', 'last_peered': '2024-09-11T05:05:15.185818+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:15.185818+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14816416, 'num_objects': 1809, 'num_object_clones': 0, 'num_object_copies': 7236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10605, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -452}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10659", 'reported_seq': 10782, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.393474+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:12.393474+0000', 'last_peered': '2024-09-11T05:05:12.393474+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:12.393474+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15332512, 'num_objects': 1872, 'num_object_clones': 0, 'num_object_copies': 7488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10657, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -349}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10782", 'reported_seq': 10887, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.687400+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:12.687400+0000', 'last_peered': '2024-09-11T05:05:12.687400+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:12.687400+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15220736, 'num_objects': 1858, 'num_object_clones': 0, 'num_object_copies': 7432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10782, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -436}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:17.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10797", 'reported_seq': 10890, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:14.935482+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:14.935482+0000', 'last_peered': '2024-09-11T05:05:14.935482+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:14.935482+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15832224, 'num_objects': 1933, 'num_object_clones': 0, 'num_object_copies': 7732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10795, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -374}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10856", 'reported_seq': 10979, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.575314+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:12.575314+0000', 'last_peered': '2024-09-11T05:05:12.575314+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:12.575314+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15821120, 'num_objects': 1932, 'num_object_clones': 0, 'num_object_copies': 7728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10853, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -434}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10667", 'reported_seq': 10779, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.865089+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:12.865089+0000', 'last_peered': '2024-09-11T05:05:12.865089+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:12.865089+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15242400, 'num_objects': 1861, 'num_object_clones': 0, 'num_object_copies': 7444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10665, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -445}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10780", 'reported_seq': 10897, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.560107+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:12.560107+0000', 'last_peered': '2024-09-11T05:05:12.560107+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:12.560107+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15339248, 'num_objects': 1873, 'num_object_clones': 0, 'num_object_copies': 7492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10777, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -425}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10853", 'reported_seq': 13641, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.926963+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:12.926963+0000', 'last_peered': '2024-09-11T05:05:12.926963+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:12.926963+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15317584, 'num_objects': 1870, 'num_object_clones': 0, 'num_object_copies': 7480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10852, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1352, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:17.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10726", 'reported_seq': 10833, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.516265+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:12.516265+0000', 'last_peered': '2024-09-11T05:05:12.516265+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:12.516265+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15283360, 'num_objects': 1866, 'num_object_clones': 0, 'num_object_copies': 7464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10724, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -398}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:17.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:17.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10640", 'reported_seq': 10759, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:12.797917+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:12.797917+0000', 'last_peered': '2024-09-11T05:05:12.797917+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:12.797917+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14488736, 'num_objects': 1769, 'num_object_clones': 0, 'num_object_copies': 7076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10638, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:17.523 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:17.523 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10901", 'reported_seq': 11021, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:15.064299+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:15.064299+0000', 'last_peered': '2024-09-11T05:05:15.064299+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:15.064299+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15365280, 'num_objects': 1876, 'num_object_clones': 0, 'num_object_copies': 7504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10899, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -412}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:17.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:18.282 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:18.282 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30673/118352 objects degraded (25.917%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":645,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":29588,"data_bytes":242340328,"bytes_used":2854162432,"bytes_avail":383692894208,"bytes_total":386547056640,"degraded_objects":30673,"degraded_total":118352,"degraded_ratio":0.25916756793294576,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:18.306 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:18.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:18.951 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:18.989 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-11T05:05:17.861918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242340328,"num_objects":29588,"num_object_clones":0,"num_object_copies":118352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29588,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156315,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2787268,"kb_used_data":477308,"kb_used_omap":14,"kb_used_meta":2309873,"kb_avail":374700092,"statfs":{"total":386547056640,"available":383692894208,"internally_reserved":0,"allocated":488763392,"data_stored":488006584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2365310518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,9,20,12,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18808832,"num_objects":-2296,"num_object_clones":0,"num_object_copies":-9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008368"},"pg_stats":[{"pgid":"1.f","version":"35'10658","reported_seq":10753,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.518124+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:17.518124+0000","last_peered":"2024-09-11T05:05:17.518124+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:17.518124+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10834","reported_seq":10949,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.052365+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:15.052365+0000","last_peered":"2024-09-11T05:05:15.052365+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:15.052365+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15594656,"num_objects":1904,"num_object_clones":0,"num_object_copies":7616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-354}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10302","reported_seq":10407,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.775293+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:12.775293+0000","last_peered":"2024-09-11T05:05:12.775293+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:12.775293+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15219304,"num_objects":1859,"num_object_clones":0,"num_object_copies":7436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10301,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10607","reported_seq":10714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.185818+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:15.185818+0000","last_peered":"2024-09-11T05:05:15.185818+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:15.185818+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816416,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-452}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10719","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.599025+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:17.599025+0000","last_peered":"2024-09-11T05:05:17.599025+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:17.599025+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14840992,"num_objects":1812,"num_object_clones":0,"num_object_copies":7248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10842","reported_seq":10948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.646366+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:17.646366+0000","last_peered":"2024-09-11T05:05:17.646366+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:17.646366+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14729216,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10797","reported_seq":10890,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:14.935482+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:14.935482+0000","last_peered":"2024-09-11T05:05:14.935482+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:14.935482+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15832224,"num_objects":1933,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10795,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-374}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10943","reported_seq":15801,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:13.084588+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:13.084588+0000","last_peered":"2024-09-11T05:05:13.084588+0000","last_clean":"2024-09-11T05:05:13.084588+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:13.084588+0000","last_undegraded":"2024-09-11T05:05:13.084588+0000","last_fullsized":"2024-09-11T05:05:13.084588+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1653,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10927","reported_seq":11050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.658010+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:17.658010+0000","last_peered":"2024-09-11T05:05:17.658010+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:17.658010+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15239488,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10667","reported_seq":10779,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.865089+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:12.865089+0000","last_peered":"2024-09-11T05:05:12.865089+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:12.865089+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15242400,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-445}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10849","reported_seq":10966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.658429+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:17.658429+0000","last_peered":"2024-09-11T05:05:17.658429+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:17.658429+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14774000,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10846,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-494}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10499","reported_seq":15100,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:13.134241+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:13.134241+0000","last_peered":"2024-09-11T05:05:13.134241+0000","last_clean":"2024-09-11T05:05:13.134241+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:13.134241+0000","last_undegraded":"2024-09-11T05:05:13.134241+0000","last_fullsized":"2024-09-11T05:05:13.134241+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15441920,"num_objects":1885,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10853","reported_seq":13641,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.926963+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:12.926963+0000","last_peered":"2024-09-11T05:05:12.926963+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:12.926963+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15317584,"num_objects":1870,"num_object_clones":0,"num_object_copies":7480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":888}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10781","reported_seq":10888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.562426+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:17.562426+0000","last_peered":"2024-09-11T05:05:17.562426+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:17.562426+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14832800,"num_objects":1811,"num_object_clones":0,"num_object_copies":7244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-453}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10640","reported_seq":10759,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:12.797917+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:12.797917+0000","last_peered":"2024-09-11T05:05:12.797917+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:12.797917+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488736,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-534}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10901","reported_seq":11021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:15.064299+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:15.064299+0000","last_peered":"2024-09-11T05:05:15.064299+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:15.064299+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15365280,"num_objects":1876,"num_object_clones":0,"num_object_copies":7504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10899,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242340328,"num_objects":29588,"num_object_clones":0,"num_object_copies":118352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29588,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156315,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":486154240,"data_stored":486154240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748176,"kb_used_data":116728,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93623664,"statfs":{"total":96636764160,"available":95870631936,"internally_reserved":0,"allocated":119529472,"data_stored":119340270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,7,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725328,"kb_used_data":120768,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93646512,"statfs":{"total":96636764160,"available":95894028288,"internally_reserved":0,"allocated":123666432,"data_stored":123477230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589748,"kb_used_data":120804,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93782092,"statfs":{"total":96636764160,"available":96032862208,"internally_reserved":0,"allocated":123703296,"data_stored":123514094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":480177874},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724016,"kb_used_data":119008,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93647824,"statfs":{"total":96636764160,"available":95895371776,"internally_reserved":0,"allocated":121864192,"data_stored":121674990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619507407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121135104,"data_stored":121135104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122822656,"data_stored":122822656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123006976,"data_stored":123006976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":119189504,"data_stored":119189504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:21.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:22.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:22.588 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-11T05:05:21.863704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237228520,"num_objects":28964,"num_object_clones":0,"num_object_copies":115856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156939,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160843,"ondisk_log_size":160843,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788168,"kb_used_data":465680,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374699192,"statfs":{"total":386547056640,"available":383691972608,"internally_reserved":0,"allocated":476856320,"data_stored":476099512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2378090038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,13,17,19,8,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17899520,"num_objects":-2185,"num_object_clones":0,"num_object_copies":-8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008546"},"pg_stats":[{"pgid":"1.f","version":"35'10658","reported_seq":10753,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.518124+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:17.518124+0000","last_peered":"2024-09-11T05:05:17.518124+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:17.518124+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10896","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:19.921879+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:19.921879+0000","last_peered":"2024-09-11T05:05:19.921879+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:19.921879+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15086752,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10371","reported_seq":10476,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.969331+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:17.969331+0000","last_peered":"2024-09-11T05:05:17.969331+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:17.969331+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14654056,"num_objects":1790,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10370,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-421}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10664","reported_seq":10772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.170474+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:20.170474+0000","last_peered":"2024-09-11T05:05:20.170474+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:20.170474+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14349472,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10662,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-509}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10719","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.599025+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:17.599025+0000","last_peered":"2024-09-11T05:05:17.599025+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:17.599025+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14840992,"num_objects":1812,"num_object_clones":0,"num_object_copies":7248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10842","reported_seq":10948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.646366+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:17.646366+0000","last_peered":"2024-09-11T05:05:17.646366+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:17.646366+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14729216,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10860","reported_seq":10953,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.238607+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:20.238607+0000","last_peered":"2024-09-11T05:05:20.238607+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:20.238607+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15316128,"num_objects":1870,"num_object_clones":0,"num_object_copies":7480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-437}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11008","reported_seq":15866,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:18.514766+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:18.514766+0000","last_peered":"2024-09-11T05:05:18.514766+0000","last_clean":"2024-09-11T05:05:18.514766+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:18.514766+0000","last_undegraded":"2024-09-11T05:05:18.514766+0000","last_fullsized":"2024-09-11T05:05:18.514766+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14868480,"num_objects":1815,"num_object_clones":0,"num_object_copies":7260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10927","reported_seq":11050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.658010+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:17.658010+0000","last_peered":"2024-09-11T05:05:17.658010+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:17.658010+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15239488,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10736","reported_seq":10847,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.838832+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:17.838832+0000","last_peered":"2024-09-11T05:05:17.838832+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:17.838832+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14677152,"num_objects":1792,"num_object_clones":0,"num_object_copies":7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-514}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10849","reported_seq":10966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.658429+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:17.658429+0000","last_peered":"2024-09-11T05:05:17.658429+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:17.658429+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14774000,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10846,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-494}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10568","reported_seq":15166,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:18.375550+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:18.375550+0000","last_peered":"2024-09-11T05:05:18.375550+0000","last_clean":"2024-09-11T05:05:18.375550+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:18.375550+0000","last_undegraded":"2024-09-11T05:05:18.375550+0000","last_fullsized":"2024-09-11T05:05:18.375550+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":1816,"num_object_clones":0,"num_object_copies":7264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4378,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10913","reported_seq":13801,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.960232+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:17.960232+0000","last_peered":"2024-09-11T05:05:17.960232+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:17.960232+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14826064,"num_objects":1810,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10781","reported_seq":10888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.562426+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:17.562426+0000","last_peered":"2024-09-11T05:05:17.562426+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:17.562426+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14832800,"num_objects":1811,"num_object_clones":0,"num_object_copies":7244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-453}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10700","reported_seq":10818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:18.062791+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:18.062791+0000","last_peered":"2024-09-11T05:05:18.062791+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:18.062791+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13997216,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10698,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-594}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10951","reported_seq":11071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.109359+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:20.109359+0000","last_peered":"2024-09-11T05:05:20.109359+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:20.109359+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14955680,"num_objects":1826,"num_object_clones":0,"num_object_copies":7304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":237228520,"num_objects":28964,"num_object_clones":0,"num_object_copies":115856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156939,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":473989120,"data_stored":473989120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160843,"ondisk_log_size":160843,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748176,"kb_used_data":116728,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93623664,"statfs":{"total":96636764160,"available":95870631936,"internally_reserved":0,"allocated":119529472,"data_stored":119340270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,7,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725568,"kb_used_data":116840,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93646272,"statfs":{"total":96636764160,"available":95893782528,"internally_reserved":0,"allocated":119644160,"data_stored":119454958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590032,"kb_used_data":116908,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93781808,"statfs":{"total":96636764160,"available":96032571392,"internally_reserved":0,"allocated":119713792,"data_stored":119524590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484437714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,4,5,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724392,"kb_used_data":115204,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93647448,"statfs":{"total":96636764160,"available":95894986752,"internally_reserved":0,"allocated":117968896,"data_stored":117779694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":117219328,"data_stored":117219328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118652928,"data_stored":118652928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118927360,"data_stored":118927360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":119189504,"data_stored":119189504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:22.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:23.172 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:23.209 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-11T05:05:21.863704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237228520,"num_objects":28964,"num_object_clones":0,"num_object_copies":115856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156939,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160843,"ondisk_log_size":160843,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788168,"kb_used_data":465680,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374699192,"statfs":{"total":386547056640,"available":383691972608,"internally_reserved":0,"allocated":476856320,"data_stored":476099512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2378090038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,13,17,19,8,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17899520,"num_objects":-2185,"num_object_clones":0,"num_object_copies":-8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008546"},"pg_stats":[{"pgid":"1.f","version":"35'10658","reported_seq":10753,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.518124+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:17.518124+0000","last_peered":"2024-09-11T05:05:17.518124+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:17.518124+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10896","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:19.921879+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:19.921879+0000","last_peered":"2024-09-11T05:05:19.921879+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:19.921879+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15086752,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10371","reported_seq":10476,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.969331+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:17.969331+0000","last_peered":"2024-09-11T05:05:17.969331+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:17.969331+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14654056,"num_objects":1790,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10370,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-421}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10664","reported_seq":10772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.170474+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:20.170474+0000","last_peered":"2024-09-11T05:05:20.170474+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:20.170474+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14349472,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10662,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-509}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10719","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.599025+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:17.599025+0000","last_peered":"2024-09-11T05:05:17.599025+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:17.599025+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14840992,"num_objects":1812,"num_object_clones":0,"num_object_copies":7248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10842","reported_seq":10948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.646366+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:17.646366+0000","last_peered":"2024-09-11T05:05:17.646366+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:17.646366+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14729216,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10860","reported_seq":10953,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.238607+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:20.238607+0000","last_peered":"2024-09-11T05:05:20.238607+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:20.238607+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15316128,"num_objects":1870,"num_object_clones":0,"num_object_copies":7480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-437}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11008","reported_seq":15866,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:18.514766+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:18.514766+0000","last_peered":"2024-09-11T05:05:18.514766+0000","last_clean":"2024-09-11T05:05:18.514766+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:18.514766+0000","last_undegraded":"2024-09-11T05:05:18.514766+0000","last_fullsized":"2024-09-11T05:05:18.514766+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14868480,"num_objects":1815,"num_object_clones":0,"num_object_copies":7260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10927","reported_seq":11050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.658010+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:17.658010+0000","last_peered":"2024-09-11T05:05:17.658010+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:17.658010+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15239488,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10736","reported_seq":10847,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.838832+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:17.838832+0000","last_peered":"2024-09-11T05:05:17.838832+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:17.838832+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14677152,"num_objects":1792,"num_object_clones":0,"num_object_copies":7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-514}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10849","reported_seq":10966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.658429+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:17.658429+0000","last_peered":"2024-09-11T05:05:17.658429+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:17.658429+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14774000,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10846,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-494}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10568","reported_seq":15166,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:18.375550+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:18.375550+0000","last_peered":"2024-09-11T05:05:18.375550+0000","last_clean":"2024-09-11T05:05:18.375550+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:18.375550+0000","last_undegraded":"2024-09-11T05:05:18.375550+0000","last_fullsized":"2024-09-11T05:05:18.375550+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":1816,"num_object_clones":0,"num_object_copies":7264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4378,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10913","reported_seq":13801,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.960232+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:17.960232+0000","last_peered":"2024-09-11T05:05:17.960232+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:17.960232+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14826064,"num_objects":1810,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10781","reported_seq":10888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:17.562426+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:17.562426+0000","last_peered":"2024-09-11T05:05:17.562426+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:17.562426+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14832800,"num_objects":1811,"num_object_clones":0,"num_object_copies":7244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-453}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10700","reported_seq":10818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:18.062791+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:18.062791+0000","last_peered":"2024-09-11T05:05:18.062791+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:18.062791+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13997216,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10698,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-594}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10951","reported_seq":11071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.109359+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:20.109359+0000","last_peered":"2024-09-11T05:05:20.109359+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:20.109359+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14955680,"num_objects":1826,"num_object_clones":0,"num_object_copies":7304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":237228520,"num_objects":28964,"num_object_clones":0,"num_object_copies":115856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156939,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":473989120,"data_stored":473989120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160843,"ondisk_log_size":160843,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748176,"kb_used_data":116728,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93623664,"statfs":{"total":96636764160,"available":95870631936,"internally_reserved":0,"allocated":119529472,"data_stored":119340270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,7,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725568,"kb_used_data":116840,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93646272,"statfs":{"total":96636764160,"available":95893782528,"internally_reserved":0,"allocated":119644160,"data_stored":119454958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590032,"kb_used_data":116908,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93781808,"statfs":{"total":96636764160,"available":96032571392,"internally_reserved":0,"allocated":119713792,"data_stored":119524590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":484437714},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,4,5,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724392,"kb_used_data":115204,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93647448,"statfs":{"total":96636764160,"available":95894986752,"internally_reserved":0,"allocated":117968896,"data_stored":117779694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":117219328,"data_stored":117219328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118652928,"data_stored":118652928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118927360,"data_stored":118927360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":119189504,"data_stored":119189504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10658", 'reported_seq': 10753, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.518124+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:17.518124+0000', 'last_peered': '2024-09-11T05:05:17.518124+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:17.518124+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15204352, 'num_objects': 1856, 'num_object_clones': 0, 'num_object_copies': 7424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10658, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -435}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10896", 'reported_seq': 11011, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:19.921879+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:19.921879+0000', 'last_peered': '2024-09-11T05:05:19.921879+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:19.921879+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15086752, 'num_objects': 1842, 'num_object_clones': 0, 'num_object_copies': 7368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10894, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -416}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10371", 'reported_seq': 10476, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.969331+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:17.969331+0000', 'last_peered': '2024-09-11T05:05:17.969331+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:17.969331+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14654056, 'num_objects': 1790, 'num_object_clones': 0, 'num_object_copies': 7160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1790, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10370, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -421}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10664", 'reported_seq': 10772, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:20.170474+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:20.170474+0000', 'last_peered': '2024-09-11T05:05:20.170474+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:20.170474+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14349472, 'num_objects': 1752, 'num_object_clones': 0, 'num_object_copies': 7008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10662, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -509}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:23.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10719", 'reported_seq': 10842, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.599025+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:17.599025+0000', 'last_peered': '2024-09-11T05:05:17.599025+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:17.599025+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14840992, 'num_objects': 1812, 'num_object_clones': 0, 'num_object_copies': 7248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10717, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -409}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10842", 'reported_seq': 10948, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.646366+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:17.646366+0000', 'last_peered': '2024-09-11T05:05:17.646366+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:17.646366+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14729216, 'num_objects': 1798, 'num_object_clones': 0, 'num_object_copies': 7192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10842, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -496}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10860", 'reported_seq': 10953, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:20.238607+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:20.238607+0000', 'last_peered': '2024-09-11T05:05:20.238607+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:20.238607+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15316128, 'num_objects': 1870, 'num_object_clones': 0, 'num_object_copies': 7480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10858, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -437}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10927", 'reported_seq': 11050, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.658010+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:17.658010+0000', 'last_peered': '2024-09-11T05:05:17.658010+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:17.658010+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15239488, 'num_objects': 1861, 'num_object_clones': 0, 'num_object_copies': 7444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10924, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -505}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10736", 'reported_seq': 10847, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.838832+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:17.838832+0000', 'last_peered': '2024-09-11T05:05:17.838832+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:17.838832+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14677152, 'num_objects': 1792, 'num_object_clones': 0, 'num_object_copies': 7168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10734, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -514}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:23.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10849", 'reported_seq': 10966, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.658429+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:17.658429+0000', 'last_peered': '2024-09-11T05:05:17.658429+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:17.658429+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14774000, 'num_objects': 1804, 'num_object_clones': 0, 'num_object_copies': 7216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10846, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -494}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:23.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:23.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10913", 'reported_seq': 13801, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.960232+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:17.960232+0000', 'last_peered': '2024-09-11T05:05:17.960232+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:17.960232+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14826064, 'num_objects': 1810, 'num_object_clones': 0, 'num_object_copies': 7240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 932, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10912, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1402, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 878}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:23.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:23.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10781", 'reported_seq': 10888, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:17.562426+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:17.562426+0000', 'last_peered': '2024-09-11T05:05:17.562426+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:17.562426+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14832800, 'num_objects': 1811, 'num_object_clones': 0, 'num_object_copies': 7244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10779, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:23.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:23.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10700", 'reported_seq': 10818, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:18.062791+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:18.062791+0000', 'last_peered': '2024-09-11T05:05:18.062791+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:18.062791+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13997216, 'num_objects': 1709, 'num_object_clones': 0, 'num_object_copies': 6836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10698, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:23.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:23.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10951", 'reported_seq': 11071, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:20.109359+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:20.109359+0000', 'last_peered': '2024-09-11T05:05:20.109359+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:20.109359+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14955680, 'num_objects': 1826, 'num_object_clones': 0, 'num_object_copies': 7304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10949, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:23.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:23.940 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30623/115856 objects degraded (26.432%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":650,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28964,"data_bytes":237228520,"bytes_used":2855084032,"bytes_avail":383691972608,"bytes_total":386547056640,"degraded_objects":30623,"degraded_total":115856,"degraded_ratio":0.26431950006905125,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:23.963 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:23.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:24.642 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:24.669 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-11T05:05:23.864364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231158248,"num_objects":28223,"num_object_clones":0,"num_object_copies":112892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157680,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1759,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793916,"kb_used_data":453404,"kb_used_omap":14,"kb_used_meta":2340401,"kb_avail":374693444,"statfs":{"total":386547056640,"available":383686086656,"internally_reserved":0,"allocated":464285696,"data_stored":463528888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2396571190},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,8,11,16,16,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21979136,"num_objects":-2683,"num_object_clones":0,"num_object_copies":-10732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2683,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008308"},"pg_stats":[{"pgid":"1.f","version":"35'10718","reported_seq":10813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.578680+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:22.578680+0000","last_peered":"2024-09-11T05:05:22.578680+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:22.578680+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14712832,"num_objects":1796,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10896","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:19.921879+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:19.921879+0000","last_peered":"2024-09-11T05:05:19.921879+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:19.921879+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15086752,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10440","reported_seq":10544,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.104650+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:23.104650+0000","last_peered":"2024-09-11T05:05:23.104650+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:23.104650+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14088808,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10439,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10664","reported_seq":10772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.170474+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:20.170474+0000","last_peered":"2024-09-11T05:05:20.170474+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:20.170474+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14349472,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10662,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-509}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10776","reported_seq":10899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.740855+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:22.740855+0000","last_peered":"2024-09-11T05:05:22.740855+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:22.740855+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14374048,"num_objects":1755,"num_object_clones":0,"num_object_copies":7020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10908","reported_seq":11014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.662079+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:22.662079+0000","last_peered":"2024-09-11T05:05:22.662079+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:22.662079+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14188544,"num_objects":1732,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10908,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10860","reported_seq":10953,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.238607+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:20.238607+0000","last_peered":"2024-09-11T05:05:20.238607+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:20.238607+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15316128,"num_objects":1870,"num_object_clones":0,"num_object_copies":7480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-437}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11056","reported_seq":15914,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:23.322045+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:23.322045+0000","last_peered":"2024-09-11T05:05:23.322045+0000","last_clean":"2024-09-11T05:05:23.322045+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:23.322045+0000","last_undegraded":"2024-09-11T05:05:23.322045+0000","last_fullsized":"2024-09-11T05:05:23.322045+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14475264,"num_objects":1767,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1766,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10991","reported_seq":11114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.750001+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:22.750001+0000","last_peered":"2024-09-11T05:05:22.750001+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:22.750001+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14715200,"num_objects":1797,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10804","reported_seq":10916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.045800+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:23.045800+0000","last_peered":"2024-09-11T05:05:23.045800+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:23.045800+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14120096,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10802,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10905","reported_seq":11021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.643089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:22.643089+0000","last_peered":"2024-09-11T05:05:22.643089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:22.643089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14315248,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10640","reported_seq":15240,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:23.535093+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:23.535093+0000","last_peered":"2024-09-11T05:05:23.535093+0000","last_clean":"2024-09-11T05:05:23.535093+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:23.535093+0000","last_undegraded":"2024-09-11T05:05:23.535093+0000","last_fullsized":"2024-09-11T05:05:23.535093+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14286848,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4450,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10989","reported_seq":14024,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.164239+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:23.164239+0000","last_peered":"2024-09-11T05:05:23.164239+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:23.164239+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14203472,"num_objects":1734,"num_object_clones":0,"num_object_copies":6936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10830","reported_seq":10937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.502448+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:22.502448+0000","last_peered":"2024-09-11T05:05:22.502448+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:22.502448+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431392,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10756","reported_seq":10875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.992170+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:22.992170+0000","last_peered":"2024-09-11T05:05:22.992170+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:22.992170+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13538464,"num_objects":1653,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-650}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10951","reported_seq":11071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:20.109359+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:20.109359+0000","last_peered":"2024-09-11T05:05:20.109359+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:20.109359+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14955680,"num_objects":1826,"num_object_clones":0,"num_object_copies":7304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":231158248,"num_objects":28223,"num_object_clones":0,"num_object_copies":112892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157680,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1759,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462073856,"data_stored":462073856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748476,"kb_used_data":112876,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93623364,"statfs":{"total":96636764160,"available":95870324736,"internally_reserved":0,"allocated":115585024,"data_stored":115395822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725868,"kb_used_data":112984,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93645972,"statfs":{"total":96636764160,"available":95893475328,"internally_reserved":0,"allocated":115695616,"data_stored":115506414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595180,"kb_used_data":112340,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93776660,"statfs":{"total":96636764160,"available":96027299840,"internally_reserved":0,"allocated":115036160,"data_stored":114846958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494399186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,6,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724392,"kb_used_data":115204,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93647448,"statfs":{"total":96636764160,"available":95894986752,"internally_reserved":0,"allocated":117968896,"data_stored":117779694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":117219328,"data_stored":117219328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114671616,"data_stored":114671616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114954240,"data_stored":114954240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115228672,"data_stored":115228672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:27.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:28.301 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:28.302 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:28.332 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-11T05:05:27.867076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229102056,"num_objects":27972,"num_object_clones":0,"num_object_copies":111888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157931,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1759,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160635,"ondisk_log_size":160635,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2794400,"kb_used_data":449728,"kb_used_omap":14,"kb_used_meta":2344561,"kb_avail":374692960,"statfs":{"total":386547056640,"available":383685591040,"internally_reserved":0,"allocated":460521472,"data_stored":459764664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2400831030},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,6,8,22,17,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16236544,"num_objects":-1982,"num_object_clones":0,"num_object_copies":-7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009376"},"pg_stats":[{"pgid":"1.f","version":"35'10718","reported_seq":10813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.578680+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:22.578680+0000","last_peered":"2024-09-11T05:05:22.578680+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:22.578680+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14712832,"num_objects":1796,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10951","reported_seq":11064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.265908+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:25.265908+0000","last_peered":"2024-09-11T05:05:25.265908+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:25.265908+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14636192,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10440","reported_seq":10544,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.104650+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:23.104650+0000","last_peered":"2024-09-11T05:05:23.104650+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:23.104650+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14088808,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10439,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10734","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.324647+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:25.324647+0000","last_peered":"2024-09-11T05:05:25.324647+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:25.324647+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13776032,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10732,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-579}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10776","reported_seq":10899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.740855+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:22.740855+0000","last_peered":"2024-09-11T05:05:22.740855+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:22.740855+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14374048,"num_objects":1755,"num_object_clones":0,"num_object_copies":7020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10908","reported_seq":11014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.662079+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:22.662079+0000","last_peered":"2024-09-11T05:05:22.662079+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:22.662079+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14188544,"num_objects":1732,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10908,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10917","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.251797+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:25.251797+0000","last_peered":"2024-09-11T05:05:25.251797+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:25.251797+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14849184,"num_objects":1813,"num_object_clones":0,"num_object_copies":7252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10915,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-494}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11056","reported_seq":15914,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:23.322045+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:23.322045+0000","last_peered":"2024-09-11T05:05:23.322045+0000","last_clean":"2024-09-11T05:05:23.322045+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:23.322045+0000","last_undegraded":"2024-09-11T05:05:23.322045+0000","last_fullsized":"2024-09-11T05:05:23.322045+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14475264,"num_objects":1767,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1766,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10991","reported_seq":11114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.750001+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:22.750001+0000","last_peered":"2024-09-11T05:05:22.750001+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:22.750001+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14715200,"num_objects":1797,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10804","reported_seq":10916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.045800+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:23.045800+0000","last_peered":"2024-09-11T05:05:23.045800+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:23.045800+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14120096,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10802,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10905","reported_seq":11021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.643089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:22.643089+0000","last_peered":"2024-09-11T05:05:22.643089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:22.643089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14315248,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10640","reported_seq":15240,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:23.535093+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:23.535093+0000","last_peered":"2024-09-11T05:05:23.535093+0000","last_clean":"2024-09-11T05:05:23.535093+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:23.535093+0000","last_undegraded":"2024-09-11T05:05:23.535093+0000","last_fullsized":"2024-09-11T05:05:23.535093+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14286848,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4450,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10989","reported_seq":14024,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.164239+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:23.164239+0000","last_peered":"2024-09-11T05:05:23.164239+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:23.164239+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14203472,"num_objects":1734,"num_object_clones":0,"num_object_copies":6936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10830","reported_seq":10937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.502448+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:22.502448+0000","last_peered":"2024-09-11T05:05:22.502448+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:22.502448+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431392,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10756","reported_seq":10875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.992170+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:22.992170+0000","last_peered":"2024-09-11T05:05:22.992170+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:22.992170+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13538464,"num_objects":1653,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-650}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11020","reported_seq":11140,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.178641+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:25.178641+0000","last_peered":"2024-09-11T05:05:25.178641+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:25.178641+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14390432,"num_objects":1757,"num_object_clones":0,"num_object_copies":7028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":229102056,"num_objects":27972,"num_object_clones":0,"num_object_copies":111888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157931,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1759,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458108928,"data_stored":458108928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160635,"ondisk_log_size":160635,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748476,"kb_used_data":112876,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93623364,"statfs":{"total":96636764160,"available":95870324736,"internally_reserved":0,"allocated":115585024,"data_stored":115395822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725868,"kb_used_data":112984,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93645972,"statfs":{"total":96636764160,"available":95893475328,"internally_reserved":0,"allocated":115695616,"data_stored":115506414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595180,"kb_used_data":112340,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93776660,"statfs":{"total":96636764160,"available":96027299840,"internally_reserved":0,"allocated":115036160,"data_stored":114846958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494399186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,6,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724876,"kb_used_data":111528,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93646964,"statfs":{"total":96636764160,"available":95894491136,"internally_reserved":0,"allocated":114204672,"data_stored":114015470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113254400,"data_stored":113254400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114671616,"data_stored":114671616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114954240,"data_stored":114954240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115228672,"data_stored":115228672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:28.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:28.946 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:28.947 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:28.967 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-11T05:05:27.867076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229102056,"num_objects":27972,"num_object_clones":0,"num_object_copies":111888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157931,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1759,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160635,"ondisk_log_size":160635,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2794400,"kb_used_data":449728,"kb_used_omap":14,"kb_used_meta":2344561,"kb_avail":374692960,"statfs":{"total":386547056640,"available":383685591040,"internally_reserved":0,"allocated":460521472,"data_stored":459764664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2400831030},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,6,8,22,17,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16236544,"num_objects":-1982,"num_object_clones":0,"num_object_copies":-7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009376"},"pg_stats":[{"pgid":"1.f","version":"35'10718","reported_seq":10813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.578680+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:22.578680+0000","last_peered":"2024-09-11T05:05:22.578680+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:22.578680+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14712832,"num_objects":1796,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-495}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10951","reported_seq":11064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.265908+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:25.265908+0000","last_peered":"2024-09-11T05:05:25.265908+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:25.265908+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14636192,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10440","reported_seq":10544,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.104650+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:23.104650+0000","last_peered":"2024-09-11T05:05:23.104650+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:23.104650+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14088808,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10439,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10734","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.324647+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:25.324647+0000","last_peered":"2024-09-11T05:05:25.324647+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:25.324647+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13776032,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10732,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-579}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10776","reported_seq":10899,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.740855+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:22.740855+0000","last_peered":"2024-09-11T05:05:22.740855+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:22.740855+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14374048,"num_objects":1755,"num_object_clones":0,"num_object_copies":7020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10908","reported_seq":11014,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.662079+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:22.662079+0000","last_peered":"2024-09-11T05:05:22.662079+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:22.662079+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14188544,"num_objects":1732,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10908,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10917","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.251797+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:25.251797+0000","last_peered":"2024-09-11T05:05:25.251797+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:25.251797+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14849184,"num_objects":1813,"num_object_clones":0,"num_object_copies":7252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10915,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-494}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11056","reported_seq":15914,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:23.322045+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:23.322045+0000","last_peered":"2024-09-11T05:05:23.322045+0000","last_clean":"2024-09-11T05:05:23.322045+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:23.322045+0000","last_undegraded":"2024-09-11T05:05:23.322045+0000","last_fullsized":"2024-09-11T05:05:23.322045+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14475264,"num_objects":1767,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1766,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10991","reported_seq":11114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.750001+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:22.750001+0000","last_peered":"2024-09-11T05:05:22.750001+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:22.750001+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14715200,"num_objects":1797,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-569}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10804","reported_seq":10916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.045800+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:23.045800+0000","last_peered":"2024-09-11T05:05:23.045800+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:23.045800+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14120096,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10802,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10905","reported_seq":11021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.643089+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:22.643089+0000","last_peered":"2024-09-11T05:05:22.643089+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:22.643089+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14315248,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10640","reported_seq":15240,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:23.535093+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:23.535093+0000","last_peered":"2024-09-11T05:05:23.535093+0000","last_clean":"2024-09-11T05:05:23.535093+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:23.535093+0000","last_undegraded":"2024-09-11T05:05:23.535093+0000","last_fullsized":"2024-09-11T05:05:23.535093+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14286848,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4450,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10989","reported_seq":14024,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:23.164239+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:23.164239+0000","last_peered":"2024-09-11T05:05:23.164239+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:23.164239+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14203472,"num_objects":1734,"num_object_clones":0,"num_object_copies":6936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10830","reported_seq":10937,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.502448+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:22.502448+0000","last_peered":"2024-09-11T05:05:22.502448+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:22.502448+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431392,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10756","reported_seq":10875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:22.992170+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:22.992170+0000","last_peered":"2024-09-11T05:05:22.992170+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:22.992170+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13538464,"num_objects":1653,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-650}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11020","reported_seq":11140,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.178641+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:25.178641+0000","last_peered":"2024-09-11T05:05:25.178641+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:25.178641+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14390432,"num_objects":1757,"num_object_clones":0,"num_object_copies":7028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":229102056,"num_objects":27972,"num_object_clones":0,"num_object_copies":111888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157931,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1759,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458108928,"data_stored":458108928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160635,"ondisk_log_size":160635,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748476,"kb_used_data":112876,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93623364,"statfs":{"total":96636764160,"available":95870324736,"internally_reserved":0,"allocated":115585024,"data_stored":115395822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725868,"kb_used_data":112984,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93645972,"statfs":{"total":96636764160,"available":95893475328,"internally_reserved":0,"allocated":115695616,"data_stored":115506414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595180,"kb_used_data":112340,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93776660,"statfs":{"total":96636764160,"available":96027299840,"internally_reserved":0,"allocated":115036160,"data_stored":114846958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":494399186},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,6,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724876,"kb_used_data":111528,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93646964,"statfs":{"total":96636764160,"available":95894491136,"internally_reserved":0,"allocated":114204672,"data_stored":114015470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113254400,"data_stored":113254400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114671616,"data_stored":114671616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114954240,"data_stored":114954240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115228672,"data_stored":115228672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:28.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:28.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10718", 'reported_seq': 10813, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:22.578680+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:22.578680+0000', 'last_peered': '2024-09-11T05:05:22.578680+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:22.578680+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14712832, 'num_objects': 1796, 'num_object_clones': 0, 'num_object_copies': 7184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10718, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -495}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10951", 'reported_seq': 11064, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:25.265908+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:25.265908+0000', 'last_peered': '2024-09-11T05:05:25.265908+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:25.265908+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14636192, 'num_objects': 1787, 'num_object_clones': 0, 'num_object_copies': 7148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10949, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10440", 'reported_seq': 10544, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:23.104650+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:23.104650+0000', 'last_peered': '2024-09-11T05:05:23.104650+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:23.104650+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14088808, 'num_objects': 1721, 'num_object_clones': 0, 'num_object_copies': 6884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1721, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10439, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -490}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10734", 'reported_seq': 10842, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:25.324647+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:25.324647+0000', 'last_peered': '2024-09-11T05:05:25.324647+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:25.324647+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13776032, 'num_objects': 1682, 'num_object_clones': 0, 'num_object_copies': 6728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10732, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -579}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10776", 'reported_seq': 10899, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:22.740855+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:22.740855+0000', 'last_peered': '2024-09-11T05:05:22.740855+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:22.740855+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14374048, 'num_objects': 1755, 'num_object_clones': 0, 'num_object_copies': 7020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10774, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -466}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10908", 'reported_seq': 11014, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:22.662079+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:22.662079+0000', 'last_peered': '2024-09-11T05:05:22.662079+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:22.662079+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14188544, 'num_objects': 1732, 'num_object_clones': 0, 'num_object_copies': 6928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10908, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -562}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10917", 'reported_seq': 11011, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:25.251797+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:25.251797+0000', 'last_peered': '2024-09-11T05:05:25.251797+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:25.251797+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14849184, 'num_objects': 1813, 'num_object_clones': 0, 'num_object_copies': 7252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10915, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -494}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10991", 'reported_seq': 11114, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:22.750001+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:22.750001+0000', 'last_peered': '2024-09-11T05:05:22.750001+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:22.750001+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14715200, 'num_objects': 1797, 'num_object_clones': 0, 'num_object_copies': 7188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10988, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -569}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10804", 'reported_seq': 10916, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:23.045800+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:23.045800+0000', 'last_peered': '2024-09-11T05:05:23.045800+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:23.045800+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14120096, 'num_objects': 1724, 'num_object_clones': 0, 'num_object_copies': 6896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10802, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -582}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10905", 'reported_seq': 11021, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:22.643089+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:22.643089+0000', 'last_peered': '2024-09-11T05:05:22.643089+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:22.643089+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14315248, 'num_objects': 1748, 'num_object_clones': 0, 'num_object_copies': 6992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10902, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -550}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:28.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10989", 'reported_seq': 14024, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:23.164239+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:23.164239+0000', 'last_peered': '2024-09-11T05:05:23.164239+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:23.164239+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14203472, 'num_objects': 1734, 'num_object_clones': 0, 'num_object_copies': 6936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 858, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10988, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1476, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:28.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:28.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10830", 'reported_seq': 10937, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:22.502448+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:22.502448+0000', 'last_peered': '2024-09-11T05:05:22.502448+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:22.502448+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14431392, 'num_objects': 1762, 'num_object_clones': 0, 'num_object_copies': 7048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10828, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:28.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:28.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10756", 'reported_seq': 10875, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:22.992170+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:22.992170+0000', 'last_peered': '2024-09-11T05:05:22.992170+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:22.992170+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13538464, 'num_objects': 1653, 'num_object_clones': 0, 'num_object_copies': 6612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10754, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:28.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:28.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11020", 'reported_seq': 11140, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:25.178641+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:25.178641+0000', 'last_peered': '2024-09-11T05:05:25.178641+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:25.178641+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14390432, 'num_objects': 1757, 'num_object_clones': 0, 'num_object_copies': 7028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11018, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:28.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:29.785 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:29.785 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30549/111888 objects degraded (27.303%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":656,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":27972,"data_bytes":229102056,"bytes_used":2861465600,"bytes_avail":383685591040,"bytes_total":386547056640,"degraded_objects":30549,"degraded_total":111888,"degraded_ratio":0.27303196053196055,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":165},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:29.806 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:29.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:30.470 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:30.471 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:30.506 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-11T05:05:29.867895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":223130088,"num_objects":27243,"num_object_clones":0,"num_object_copies":108972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158660,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1814,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160764,"ondisk_log_size":160764,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778464,"kb_used_data":437960,"kb_used_omap":14,"kb_used_meta":2340401,"kb_avail":374708896,"statfs":{"total":386547056640,"available":383701909504,"internally_reserved":0,"allocated":448471040,"data_stored":447714232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2396571190},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,1,8,6,21,21,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19210240,"num_objects":-2345,"num_object_clones":0,"num_object_copies":-9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2345,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005976"},"pg_stats":[{"pgid":"1.f","version":"35'10774","reported_seq":10869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:27.784914+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:27.784914+0000","last_peered":"2024-09-11T05:05:27.784914+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:27.784914+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14254080,"num_objects":1740,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10951","reported_seq":11064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.265908+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:25.265908+0000","last_peered":"2024-09-11T05:05:25.265908+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:25.265908+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14636192,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10495","reported_seq":10600,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:27.977134+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:27.977134+0000","last_peered":"2024-09-11T05:05:27.977134+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:27.977134+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13638248,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10494,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10734","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.324647+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:25.324647+0000","last_peered":"2024-09-11T05:05:25.324647+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:25.324647+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13776032,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10732,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-579}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10837","reported_seq":10960,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:27.693223+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:27.693223+0000","last_peered":"2024-09-11T05:05:27.693223+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:27.693223+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13874336,"num_objects":1694,"num_object_clones":0,"num_object_copies":6776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10835,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-527}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10967","reported_seq":11069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:27.743062+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:27.743062+0000","last_peered":"2024-09-11T05:05:27.743062+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:27.743062+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13705216,"num_objects":1673,"num_object_clones":0,"num_object_copies":6692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10967,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10917","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.251797+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:25.251797+0000","last_peered":"2024-09-11T05:05:25.251797+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:25.251797+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14849184,"num_objects":1813,"num_object_clones":0,"num_object_copies":7252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10915,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-494}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11112","reported_seq":15970,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:28.678256+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:28.678256+0000","last_peered":"2024-09-11T05:05:28.678256+0000","last_clean":"2024-09-11T05:05:28.678256+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:28.678256+0000","last_undegraded":"2024-09-11T05:05:28.678256+0000","last_fullsized":"2024-09-11T05:05:28.678256+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14016512,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1822,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11059","reported_seq":11181,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:27.765334+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:27.765334+0000","last_peered":"2024-09-11T05:05:27.765334+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:27.765334+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14158144,"num_objects":1729,"num_object_clones":0,"num_object_copies":6916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11056,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10866","reported_seq":10978,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:28.246362+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:28.246362+0000","last_peered":"2024-09-11T05:05:28.246362+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:28.246362+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13612192,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10864,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-644}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10968","reported_seq":11084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:27.855869+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:27.855869+0000","last_peered":"2024-09-11T05:05:27.855869+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:27.855869+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13799152,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-613}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10705","reported_seq":15306,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:28.611310+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:28.611310+0000","last_peered":"2024-09-11T05:05:28.611310+0000","last_clean":"2024-09-11T05:05:28.611310+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:28.611310+0000","last_undegraded":"2024-09-11T05:05:28.611310+0000","last_fullsized":"2024-09-11T05:05:28.611310+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13754368,"num_objects":1679,"num_object_clones":0,"num_object_copies":6716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4515,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11054","reported_seq":14200,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:28.231710+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:28.231710+0000","last_peered":"2024-09-11T05:05:28.231710+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:28.231710+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13670992,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10874","reported_seq":10981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:27.695920+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:27.695920+0000","last_peered":"2024-09-11T05:05:27.695920+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:27.695920+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14070944,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10872,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-546}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10831","reported_seq":10948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:27.701248+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:27.701248+0000","last_peered":"2024-09-11T05:05:27.701248+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:27.701248+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12924064,"num_objects":1578,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-725}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11020","reported_seq":11140,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:25.178641+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:25.178641+0000","last_peered":"2024-09-11T05:05:25.178641+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:25.178641+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14390432,"num_objects":1757,"num_object_clones":0,"num_object_copies":7028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":223130088,"num_objects":27243,"num_object_clones":0,"num_object_copies":108972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158660,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1814,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445820928,"data_stored":445820928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160764,"ondisk_log_size":160764,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732080,"kb_used_data":108956,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93639760,"statfs":{"total":96636764160,"available":95887114240,"internally_reserved":0,"allocated":111570944,"data_stored":111381742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":638054096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,6,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726060,"kb_used_data":109020,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93645780,"statfs":{"total":96636764160,"available":95893278720,"internally_reserved":0,"allocated":111636480,"data_stored":111447278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595448,"kb_used_data":108456,"kb_used_omap":4,"kb_used_meta":486971,"kb_avail":93776392,"statfs":{"total":96636764160,"available":96027025408,"internally_reserved":0,"allocated":111058944,"data_stored":110869742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":498659026},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,0,6,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724876,"kb_used_data":111528,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93646964,"statfs":{"total":96636764160,"available":95894491136,"internally_reserved":0,"allocated":114204672,"data_stored":114015470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113254400,"data_stored":113254400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110616576,"data_stored":110616576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110825472,"data_stored":110825472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111124480,"data_stored":111124480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:33.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:34.097 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:34.119 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-11T05:05:33.869577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215077352,"num_objects":26260,"num_object_clones":0,"num_object_copies":105040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26260,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159643,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1867,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2817084,"kb_used_data":422436,"kb_used_omap":14,"kb_used_meta":2394545,"kb_avail":374670276,"statfs":{"total":386547056640,"available":383662362624,"internally_reserved":0,"allocated":432574464,"data_stored":431817656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2452014646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,8,8,16,15,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22151168,"num_objects":-2704,"num_object_clones":0,"num_object_copies":-10816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2704,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005873"},"pg_stats":[{"pgid":"1.f","version":"35'10857","reported_seq":10952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.966499+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:32.966499+0000","last_peered":"2024-09-11T05:05:32.966499+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:32.966499+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13574144,"num_objects":1657,"num_object_clones":0,"num_object_copies":6628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11012","reported_seq":11127,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:30.231588+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:30.231588+0000","last_peered":"2024-09-11T05:05:30.231588+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:30.231588+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14136480,"num_objects":1726,"num_object_clones":0,"num_object_copies":6904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11010,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10555","reported_seq":10660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.200889+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:33.200889+0000","last_peered":"2024-09-11T05:05:33.200889+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:33.200889+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13146728,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10554,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10802","reported_seq":10909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:30.377924+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:30.377924+0000","last_peered":"2024-09-11T05:05:30.377924+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:30.377924+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13218976,"num_objects":1614,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-647}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10900","reported_seq":11023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.754792+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:32.754792+0000","last_peered":"2024-09-11T05:05:32.754792+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:32.754792+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13358240,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10898,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11040","reported_seq":11146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.875772+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:32.875772+0000","last_peered":"2024-09-11T05:05:32.875772+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:32.875772+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":1600,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11040,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10972","reported_seq":11066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:29.771476+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:29.771476+0000","last_peered":"2024-09-11T05:05:29.771476+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:29.771476+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14398624,"num_objects":1758,"num_object_clones":0,"num_object_copies":7032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10970,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-549}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11165","reported_seq":16023,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:33.705199+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:33.705199+0000","last_peered":"2024-09-11T05:05:33.705199+0000","last_clean":"2024-09-11T05:05:33.705199+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:33.705199+0000","last_undegraded":"2024-09-11T05:05:33.705199+0000","last_fullsized":"2024-09-11T05:05:33.705199+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13582336,"num_objects":1658,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11125","reported_seq":11248,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.425132+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:32.425132+0000","last_peered":"2024-09-11T05:05:32.425132+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:32.425132+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13617472,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-703}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10918","reported_seq":11031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.145203+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:33.145203+0000","last_peered":"2024-09-11T05:05:33.145203+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:33.145203+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13186208,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11022","reported_seq":11138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.997295+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:32.997295+0000","last_peered":"2024-09-11T05:05:32.997295+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:32.997295+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13356784,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11019,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10771","reported_seq":15372,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:33.741992+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:33.741992+0000","last_peered":"2024-09-11T05:05:33.741992+0000","last_clean":"2024-09-11T05:05:33.741992+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:33.741992+0000","last_undegraded":"2024-09-11T05:05:33.741992+0000","last_fullsized":"2024-09-11T05:05:33.741992+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13213696,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11107","reported_seq":14358,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.352141+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:33.352141+0000","last_peered":"2024-09-11T05:05:33.352141+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:33.352141+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13236816,"num_objects":1616,"num_object_clones":0,"num_object_copies":6464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":750,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10943","reported_seq":11050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.812462+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:32.812462+0000","last_peered":"2024-09-11T05:05:32.812462+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:32.812462+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13505696,"num_objects":1649,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-615}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10882","reported_seq":11001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.199238+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:33.199238+0000","last_peered":"2024-09-11T05:05:33.199238+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:33.199238+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12506272,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10880,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-776}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11076","reported_seq":11195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:30.359946+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:30.359946+0000","last_peered":"2024-09-11T05:05:30.359946+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:30.359946+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13931680,"num_objects":1701,"num_object_clones":0,"num_object_copies":6804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11074,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":215077352,"num_objects":26260,"num_object_clones":0,"num_object_copies":105040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26260,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159643,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1867,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":429719552,"data_stored":429719552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750464,"kb_used_data":105132,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93621376,"statfs":{"total":96636764160,"available":95868289024,"internally_reserved":0,"allocated":107655168,"data_stored":107465966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,6,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727800,"kb_used_data":105172,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93644040,"statfs":{"total":96636764160,"available":95891496960,"internally_reserved":0,"allocated":107696128,"data_stored":107506926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613768,"kb_used_data":104568,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93758072,"statfs":{"total":96636764160,"available":96008265728,"internally_reserved":0,"allocated":107077632,"data_stored":106888430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521400018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725052,"kb_used_data":107564,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93646788,"statfs":{"total":96636764160,"available":95894310912,"internally_reserved":0,"allocated":110145536,"data_stored":109956334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109219840,"data_stored":109219840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106582016,"data_stored":106582016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106819584,"data_stored":106819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107098112,"data_stored":107098112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:34.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:34.800 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-11T05:05:33.869577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215077352,"num_objects":26260,"num_object_clones":0,"num_object_copies":105040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26260,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159643,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1867,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2817084,"kb_used_data":422436,"kb_used_omap":14,"kb_used_meta":2394545,"kb_avail":374670276,"statfs":{"total":386547056640,"available":383662362624,"internally_reserved":0,"allocated":432574464,"data_stored":431817656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2452014646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,8,8,16,15,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22151168,"num_objects":-2704,"num_object_clones":0,"num_object_copies":-10816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2704,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005873"},"pg_stats":[{"pgid":"1.f","version":"35'10857","reported_seq":10952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.966499+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:32.966499+0000","last_peered":"2024-09-11T05:05:32.966499+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:32.966499+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13574144,"num_objects":1657,"num_object_clones":0,"num_object_copies":6628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11012","reported_seq":11127,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:30.231588+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:30.231588+0000","last_peered":"2024-09-11T05:05:30.231588+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:30.231588+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14136480,"num_objects":1726,"num_object_clones":0,"num_object_copies":6904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11010,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10555","reported_seq":10660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.200889+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:33.200889+0000","last_peered":"2024-09-11T05:05:33.200889+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:33.200889+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13146728,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10554,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10802","reported_seq":10909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:30.377924+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:30.377924+0000","last_peered":"2024-09-11T05:05:30.377924+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:30.377924+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13218976,"num_objects":1614,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-647}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10900","reported_seq":11023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.754792+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:32.754792+0000","last_peered":"2024-09-11T05:05:32.754792+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:32.754792+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13358240,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10898,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11040","reported_seq":11146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.875772+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:32.875772+0000","last_peered":"2024-09-11T05:05:32.875772+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:32.875772+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":1600,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11040,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10972","reported_seq":11066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:29.771476+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:29.771476+0000","last_peered":"2024-09-11T05:05:29.771476+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:29.771476+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14398624,"num_objects":1758,"num_object_clones":0,"num_object_copies":7032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10970,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-549}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11165","reported_seq":16023,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:33.705199+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:33.705199+0000","last_peered":"2024-09-11T05:05:33.705199+0000","last_clean":"2024-09-11T05:05:33.705199+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:33.705199+0000","last_undegraded":"2024-09-11T05:05:33.705199+0000","last_fullsized":"2024-09-11T05:05:33.705199+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13582336,"num_objects":1658,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11125","reported_seq":11248,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.425132+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:32.425132+0000","last_peered":"2024-09-11T05:05:32.425132+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:32.425132+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13617472,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-703}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10918","reported_seq":11031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.145203+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:33.145203+0000","last_peered":"2024-09-11T05:05:33.145203+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:33.145203+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13186208,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11022","reported_seq":11138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.997295+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:32.997295+0000","last_peered":"2024-09-11T05:05:32.997295+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:32.997295+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13356784,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11019,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10771","reported_seq":15372,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:33.741992+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:33.741992+0000","last_peered":"2024-09-11T05:05:33.741992+0000","last_clean":"2024-09-11T05:05:33.741992+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:33.741992+0000","last_undegraded":"2024-09-11T05:05:33.741992+0000","last_fullsized":"2024-09-11T05:05:33.741992+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13213696,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11107","reported_seq":14358,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.352141+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:33.352141+0000","last_peered":"2024-09-11T05:05:33.352141+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:33.352141+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13236816,"num_objects":1616,"num_object_clones":0,"num_object_copies":6464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":750,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10943","reported_seq":11050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.812462+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:32.812462+0000","last_peered":"2024-09-11T05:05:32.812462+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:32.812462+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13505696,"num_objects":1649,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-615}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10882","reported_seq":11001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.199238+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:33.199238+0000","last_peered":"2024-09-11T05:05:33.199238+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:33.199238+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12506272,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10880,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-776}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11076","reported_seq":11195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:30.359946+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:30.359946+0000","last_peered":"2024-09-11T05:05:30.359946+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:30.359946+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13931680,"num_objects":1701,"num_object_clones":0,"num_object_copies":6804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11074,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":215077352,"num_objects":26260,"num_object_clones":0,"num_object_copies":105040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26260,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159643,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1867,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":429719552,"data_stored":429719552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750464,"kb_used_data":105132,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93621376,"statfs":{"total":96636764160,"available":95868289024,"internally_reserved":0,"allocated":107655168,"data_stored":107465966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,6,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727800,"kb_used_data":105172,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93644040,"statfs":{"total":96636764160,"available":95891496960,"internally_reserved":0,"allocated":107696128,"data_stored":107506926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613768,"kb_used_data":104568,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93758072,"statfs":{"total":96636764160,"available":96008265728,"internally_reserved":0,"allocated":107077632,"data_stored":106888430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521400018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725052,"kb_used_data":107564,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93646788,"statfs":{"total":96636764160,"available":95894310912,"internally_reserved":0,"allocated":110145536,"data_stored":109956334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109219840,"data_stored":109219840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106582016,"data_stored":106582016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106819584,"data_stored":106819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107098112,"data_stored":107098112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:34.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:34.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10857", 'reported_seq': 10952, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:32.966499+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:32.966499+0000', 'last_peered': '2024-09-11T05:05:32.966499+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:32.966499+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13574144, 'num_objects': 1657, 'num_object_clones': 0, 'num_object_copies': 6628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10857, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -634}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:34.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:34.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11012", 'reported_seq': 11127, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:30.231588+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:30.231588+0000', 'last_peered': '2024-09-11T05:05:30.231588+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:30.231588+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14136480, 'num_objects': 1726, 'num_object_clones': 0, 'num_object_copies': 6904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11010, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:34.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:34.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10555", 'reported_seq': 10660, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:33.200889+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:33.200889+0000', 'last_peered': '2024-09-11T05:05:33.200889+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:33.200889+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13146728, 'num_objects': 1606, 'num_object_clones': 0, 'num_object_copies': 6424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1606, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10554, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -605}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:34.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10802", 'reported_seq': 10909, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:30.377924+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:30.377924+0000', 'last_peered': '2024-09-11T05:05:30.377924+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:30.377924+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13218976, 'num_objects': 1614, 'num_object_clones': 0, 'num_object_copies': 6456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10800, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -647}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10900", 'reported_seq': 11023, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:32.754792+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:32.754792+0000', 'last_peered': '2024-09-11T05:05:32.754792+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:32.754792+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13358240, 'num_objects': 1631, 'num_object_clones': 0, 'num_object_copies': 6524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10898, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -590}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11040", 'reported_seq': 11146, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:32.875772+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:32.875772+0000', 'last_peered': '2024-09-11T05:05:32.875772+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:32.875772+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13107200, 'num_objects': 1600, 'num_object_clones': 0, 'num_object_copies': 6400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11040, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -694}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10972", 'reported_seq': 11066, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:29.771476+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:29.771476+0000', 'last_peered': '2024-09-11T05:05:29.771476+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:29.771476+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14398624, 'num_objects': 1758, 'num_object_clones': 0, 'num_object_copies': 7032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10970, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -549}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11125", 'reported_seq': 11248, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:32.425132+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:32.425132+0000', 'last_peered': '2024-09-11T05:05:32.425132+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:32.425132+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13617472, 'num_objects': 1663, 'num_object_clones': 0, 'num_object_copies': 6652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11122, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -703}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:34.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10918", 'reported_seq': 11031, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:33.145203+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:33.145203+0000', 'last_peered': '2024-09-11T05:05:33.145203+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:33.145203+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13186208, 'num_objects': 1610, 'num_object_clones': 0, 'num_object_copies': 6440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10916, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -696}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11022", 'reported_seq': 11138, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:32.997295+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:32.997295+0000', 'last_peered': '2024-09-11T05:05:32.997295+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:32.997295+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13356784, 'num_objects': 1631, 'num_object_clones': 0, 'num_object_copies': 6524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11019, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -667}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11107", 'reported_seq': 14358, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:33.352141+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:33.352141+0000', 'last_peered': '2024-09-11T05:05:33.352141+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:33.352141+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13236816, 'num_objects': 1616, 'num_object_clones': 0, 'num_object_copies': 6464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 750, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11106, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1584, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 866}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10943", 'reported_seq': 11050, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:32.812462+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:32.812462+0000', 'last_peered': '2024-09-11T05:05:32.812462+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:32.812462+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13505696, 'num_objects': 1649, 'num_object_clones': 0, 'num_object_copies': 6596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10941, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10882", 'reported_seq': 11001, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:33.199238+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:33.199238+0000', 'last_peered': '2024-09-11T05:05:33.199238+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:33.199238+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12506272, 'num_objects': 1527, 'num_object_clones': 0, 'num_object_copies': 6108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10880, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:34.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11076", 'reported_seq': 11195, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:30.359946+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:30.359946+0000', 'last_peered': '2024-09-11T05:05:30.359946+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:30.359946+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13931680, 'num_objects': 1701, 'num_object_clones': 0, 'num_object_copies': 6804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11074, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -587}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:34.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:35.602 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:35.602 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30441/105040 objects degraded (28.980%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":662,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26260,"data_bytes":215077352,"bytes_used":2884694016,"bytes_avail":383662362624,"bytes_total":386547056640,"degraded_objects":30441,"degraded_total":105040,"degraded_ratio":0.28980388423457731,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":225},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:35.638 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:35.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:36.299 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:36.322 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-11T05:05:35.870451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":212972008,"num_objects":26003,"num_object_clones":0,"num_object_copies":104012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159900,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1867,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818692,"kb_used_data":418472,"kb_used_omap":14,"kb_used_meta":2400113,"kb_avail":374668668,"statfs":{"total":386547056640,"available":383660716032,"internally_reserved":0,"allocated":428515328,"data_stored":427758520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2457716278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,7,6,17,13,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18186240,"num_objects":-2220,"num_object_clones":0,"num_object_copies":-8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2220,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006086"},"pg_stats":[{"pgid":"1.f","version":"35'10857","reported_seq":10952,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.966499+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:32.966499+0000","last_peered":"2024-09-11T05:05:32.966499+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:32.966499+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13574144,"num_objects":1657,"num_object_clones":0,"num_object_copies":6628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11078","reported_seq":11193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.269212+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:35.269212+0000","last_peered":"2024-09-11T05:05:35.269212+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:35.269212+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13595808,"num_objects":1660,"num_object_clones":0,"num_object_copies":6640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10555","reported_seq":10660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.200889+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:33.200889+0000","last_peered":"2024-09-11T05:05:33.200889+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:33.200889+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13146728,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10554,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10875","reported_seq":10984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.382774+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:35.382774+0000","last_peered":"2024-09-11T05:05:35.382774+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:35.382774+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12620960,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10900","reported_seq":11023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.754792+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:32.754792+0000","last_peered":"2024-09-11T05:05:32.754792+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:32.754792+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13358240,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10898,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11040","reported_seq":11146,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.875772+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:32.875772+0000","last_peered":"2024-09-11T05:05:32.875772+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:32.875772+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":1600,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11040,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11028","reported_seq":11122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.386993+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:35.386993+0000","last_peered":"2024-09-11T05:05:35.386993+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:35.386993+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939872,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-605}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11165","reported_seq":16023,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:33.705199+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:33.705199+0000","last_peered":"2024-09-11T05:05:33.705199+0000","last_clean":"2024-09-11T05:05:33.705199+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:33.705199+0000","last_undegraded":"2024-09-11T05:05:33.705199+0000","last_fullsized":"2024-09-11T05:05:33.705199+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13582336,"num_objects":1658,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11125","reported_seq":11248,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.425132+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:32.425132+0000","last_peered":"2024-09-11T05:05:32.425132+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:32.425132+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13617472,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-703}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10918","reported_seq":11031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.145203+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:33.145203+0000","last_peered":"2024-09-11T05:05:33.145203+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:33.145203+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13186208,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11022","reported_seq":11138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.997295+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:32.997295+0000","last_peered":"2024-09-11T05:05:32.997295+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:32.997295+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13356784,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11019,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10771","reported_seq":15372,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:33.741992+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:33.741992+0000","last_peered":"2024-09-11T05:05:33.741992+0000","last_clean":"2024-09-11T05:05:33.741992+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:33.741992+0000","last_undegraded":"2024-09-11T05:05:33.741992+0000","last_fullsized":"2024-09-11T05:05:33.741992+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13213696,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11107","reported_seq":14358,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.352141+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:33.352141+0000","last_peered":"2024-09-11T05:05:33.352141+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:33.352141+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13236816,"num_objects":1616,"num_object_clones":0,"num_object_copies":6464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":750,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10943","reported_seq":11050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:32.812462+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:32.812462+0000","last_peered":"2024-09-11T05:05:32.812462+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:32.812462+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13505696,"num_objects":1649,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-615}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10882","reported_seq":11001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:33.199238+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:33.199238+0000","last_peered":"2024-09-11T05:05:33.199238+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:33.199238+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12506272,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10880,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-776}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11138","reported_seq":11258,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.240820+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:35.240820+0000","last_peered":"2024-09-11T05:05:35.240820+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:35.240820+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13423776,"num_objects":1639,"num_object_clones":0,"num_object_copies":6556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11136,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":212972008,"num_objects":26003,"num_object_clones":0,"num_object_copies":104012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159900,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1867,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425762816,"data_stored":425762816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750464,"kb_used_data":105132,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93621376,"statfs":{"total":96636764160,"available":95868289024,"internally_reserved":0,"allocated":107655168,"data_stored":107465966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,6,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727800,"kb_used_data":105172,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93644040,"statfs":{"total":96636764160,"available":95891496960,"internally_reserved":0,"allocated":107696128,"data_stored":107506926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613768,"kb_used_data":104568,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93758072,"statfs":{"total":96636764160,"available":96008265728,"internally_reserved":0,"allocated":107077632,"data_stored":106888430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521400018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726660,"kb_used_data":103600,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93645180,"statfs":{"total":96636764160,"available":95892664320,"internally_reserved":0,"allocated":106086400,"data_stored":105897198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":637988559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":105263104,"data_stored":105263104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106582016,"data_stored":106582016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106819584,"data_stored":106819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":107098112,"data_stored":107098112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:39.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:39.958 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:39.959 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:39.983 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-11T05:05:39.871845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":207114728,"num_objects":25288,"num_object_clones":0,"num_object_copies":101152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160615,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1926,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819996,"kb_used_data":407300,"kb_used_omap":14,"kb_used_meta":2412593,"kb_avail":374667364,"statfs":{"total":386547056640,"available":383659380736,"internally_reserved":0,"allocated":417075200,"data_stored":416318392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2470495798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,2,6,12,6,9,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21987328,"num_objects":-2684,"num_object_clones":0,"num_object_copies":-10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004768"},"pg_stats":[{"pgid":"1.f","version":"35'10916","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.875334+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:37.875334+0000","last_peered":"2024-09-11T05:05:37.875334+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:37.875334+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13090816,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11078","reported_seq":11193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.269212+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:35.269212+0000","last_peered":"2024-09-11T05:05:35.269212+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:35.269212+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13595808,"num_objects":1660,"num_object_clones":0,"num_object_copies":6640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10616","reported_seq":10721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.256583+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:38.256583+0000","last_peered":"2024-09-11T05:05:38.256583+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:38.256583+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12647016,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10615,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10875","reported_seq":10984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.382774+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:35.382774+0000","last_peered":"2024-09-11T05:05:35.382774+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:35.382774+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12620960,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10965","reported_seq":11088,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.052563+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:38.052563+0000","last_peered":"2024-09-11T05:05:38.052563+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:38.052563+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12825760,"num_objects":1566,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11103","reported_seq":11208,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.965148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:37.965148+0000","last_peered":"2024-09-11T05:05:37.965148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:37.965148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12591104,"num_objects":1537,"num_object_clones":0,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11103,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11028","reported_seq":11122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.386993+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:35.386993+0000","last_peered":"2024-09-11T05:05:35.386993+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:35.386993+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939872,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-605}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11234","reported_seq":16092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:38.598938+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:38.598938+0000","last_peered":"2024-09-11T05:05:38.598938+0000","last_clean":"2024-09-11T05:05:38.598938+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:38.598938+0000","last_undegraded":"2024-09-11T05:05:38.598938+0000","last_fullsized":"2024-09-11T05:05:38.598938+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13017088,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11187","reported_seq":11310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.956562+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:37.956562+0000","last_peered":"2024-09-11T05:05:37.956562+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:37.956562+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109568,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11184,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10967","reported_seq":11080,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.129384+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:38.129384+0000","last_peered":"2024-09-11T05:05:38.129384+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:38.129384+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12784800,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11074","reported_seq":11190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.005604+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:38.005604+0000","last_peered":"2024-09-11T05:05:38.005604+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:38.005604+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12930800,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10840","reported_seq":15441,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:38.598915+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:38.598915+0000","last_peered":"2024-09-11T05:05:38.598915+0000","last_clean":"2024-09-11T05:05:38.598915+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:38.598915+0000","last_undegraded":"2024-09-11T05:05:38.598915+0000","last_fullsized":"2024-09-11T05:05:38.598915+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648448,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11165","reported_seq":14532,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.361160+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:38.361160+0000","last_peered":"2024-09-11T05:05:38.361160+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:38.361160+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12761680,"num_objects":1558,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11164,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11002","reported_seq":11109,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.668679+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:37.668679+0000","last_peered":"2024-09-11T05:05:37.668679+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:37.668679+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13022368,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11000,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-674}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10931","reported_seq":11048,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.358984+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:38.358984+0000","last_peered":"2024-09-11T05:05:38.358984+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:38.358984+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12104864,"num_objects":1478,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11138","reported_seq":11258,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.240820+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:35.240820+0000","last_peered":"2024-09-11T05:05:35.240820+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:35.240820+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13423776,"num_objects":1639,"num_object_clones":0,"num_object_copies":6556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11136,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":207114728,"num_objects":25288,"num_object_clones":0,"num_object_copies":101152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160615,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1926,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":414085120,"data_stored":414085120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750920,"kb_used_data":101432,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93620920,"statfs":{"total":96636764160,"available":95867822080,"internally_reserved":0,"allocated":103866368,"data_stored":103677166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,3,1,2,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728256,"kb_used_data":101476,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93643584,"statfs":{"total":96636764160,"available":95891030016,"internally_reserved":0,"allocated":103911424,"data_stored":103722222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614160,"kb_used_data":100792,"kb_used_omap":4,"kb_used_meta":513339,"kb_avail":93757680,"statfs":{"total":96636764160,"available":96007864320,"internally_reserved":0,"allocated":103211008,"data_stored":103021806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525659858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726660,"kb_used_data":103600,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93645180,"statfs":{"total":96636764160,"available":95892664320,"internally_reserved":0,"allocated":106086400,"data_stored":105897198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":637988559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":105263104,"data_stored":105263104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102703104,"data_stored":102703104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102948864,"data_stored":102948864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":103170048,"data_stored":103170048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:39.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:40.634 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:40.661 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-11T05:05:39.871845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":207114728,"num_objects":25288,"num_object_clones":0,"num_object_copies":101152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160615,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1926,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819996,"kb_used_data":407300,"kb_used_omap":14,"kb_used_meta":2412593,"kb_avail":374667364,"statfs":{"total":386547056640,"available":383659380736,"internally_reserved":0,"allocated":417075200,"data_stored":416318392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2470495798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,2,6,12,6,9,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21987328,"num_objects":-2684,"num_object_clones":0,"num_object_copies":-10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004768"},"pg_stats":[{"pgid":"1.f","version":"35'10916","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.875334+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:37.875334+0000","last_peered":"2024-09-11T05:05:37.875334+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:37.875334+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13090816,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11078","reported_seq":11193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.269212+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:35.269212+0000","last_peered":"2024-09-11T05:05:35.269212+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:35.269212+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13595808,"num_objects":1660,"num_object_clones":0,"num_object_copies":6640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10616","reported_seq":10721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.256583+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:38.256583+0000","last_peered":"2024-09-11T05:05:38.256583+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:38.256583+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12647016,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10615,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10875","reported_seq":10984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.382774+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:35.382774+0000","last_peered":"2024-09-11T05:05:35.382774+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:35.382774+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12620960,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10965","reported_seq":11088,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.052563+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:38.052563+0000","last_peered":"2024-09-11T05:05:38.052563+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:38.052563+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12825760,"num_objects":1566,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11103","reported_seq":11208,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.965148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:37.965148+0000","last_peered":"2024-09-11T05:05:37.965148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:37.965148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12591104,"num_objects":1537,"num_object_clones":0,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11103,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11028","reported_seq":11122,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.386993+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:35.386993+0000","last_peered":"2024-09-11T05:05:35.386993+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:35.386993+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939872,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-605}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11234","reported_seq":16092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:38.598938+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:38.598938+0000","last_peered":"2024-09-11T05:05:38.598938+0000","last_clean":"2024-09-11T05:05:38.598938+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:38.598938+0000","last_undegraded":"2024-09-11T05:05:38.598938+0000","last_fullsized":"2024-09-11T05:05:38.598938+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13017088,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11187","reported_seq":11310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.956562+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:37.956562+0000","last_peered":"2024-09-11T05:05:37.956562+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:37.956562+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109568,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11184,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10967","reported_seq":11080,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.129384+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:38.129384+0000","last_peered":"2024-09-11T05:05:38.129384+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:38.129384+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12784800,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11074","reported_seq":11190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.005604+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:38.005604+0000","last_peered":"2024-09-11T05:05:38.005604+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:38.005604+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12930800,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10840","reported_seq":15441,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:38.598915+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:38.598915+0000","last_peered":"2024-09-11T05:05:38.598915+0000","last_clean":"2024-09-11T05:05:38.598915+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:38.598915+0000","last_undegraded":"2024-09-11T05:05:38.598915+0000","last_fullsized":"2024-09-11T05:05:38.598915+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648448,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11165","reported_seq":14532,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.361160+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:38.361160+0000","last_peered":"2024-09-11T05:05:38.361160+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:38.361160+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12761680,"num_objects":1558,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11164,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11002","reported_seq":11109,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.668679+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:37.668679+0000","last_peered":"2024-09-11T05:05:37.668679+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:37.668679+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13022368,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11000,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-674}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10931","reported_seq":11048,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.358984+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:38.358984+0000","last_peered":"2024-09-11T05:05:38.358984+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:38.358984+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12104864,"num_objects":1478,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11138","reported_seq":11258,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:35.240820+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:35.240820+0000","last_peered":"2024-09-11T05:05:35.240820+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:35.240820+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13423776,"num_objects":1639,"num_object_clones":0,"num_object_copies":6556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11136,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":207114728,"num_objects":25288,"num_object_clones":0,"num_object_copies":101152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160615,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1926,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":414085120,"data_stored":414085120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750920,"kb_used_data":101432,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93620920,"statfs":{"total":96636764160,"available":95867822080,"internally_reserved":0,"allocated":103866368,"data_stored":103677166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,3,1,2,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728256,"kb_used_data":101476,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93643584,"statfs":{"total":96636764160,"available":95891030016,"internally_reserved":0,"allocated":103911424,"data_stored":103722222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614160,"kb_used_data":100792,"kb_used_omap":4,"kb_used_meta":513339,"kb_avail":93757680,"statfs":{"total":96636764160,"available":96007864320,"internally_reserved":0,"allocated":103211008,"data_stored":103021806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525659858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726660,"kb_used_data":103600,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93645180,"statfs":{"total":96636764160,"available":95892664320,"internally_reserved":0,"allocated":106086400,"data_stored":105897198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":637988559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":105263104,"data_stored":105263104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102703104,"data_stored":102703104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102948864,"data_stored":102948864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":103170048,"data_stored":103170048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:40.662 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:40.662 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10916", 'reported_seq': 11011, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:37.875334+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:37.875334+0000', 'last_peered': '2024-09-11T05:05:37.875334+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:37.875334+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13090816, 'num_objects': 1598, 'num_object_clones': 0, 'num_object_copies': 6392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10916, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -693}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11078", 'reported_seq': 11193, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:35.269212+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:35.269212+0000', 'last_peered': '2024-09-11T05:05:35.269212+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:35.269212+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13595808, 'num_objects': 1660, 'num_object_clones': 0, 'num_object_copies': 6640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11076, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -598}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10616", 'reported_seq': 10721, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:38.256583+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:38.256583+0000', 'last_peered': '2024-09-11T05:05:38.256583+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:38.256583+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12647016, 'num_objects': 1545, 'num_object_clones': 0, 'num_object_copies': 6180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1545, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10615, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -666}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10875", 'reported_seq': 10984, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:35.382774+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:35.382774+0000', 'last_peered': '2024-09-11T05:05:35.382774+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:35.382774+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12620960, 'num_objects': 1541, 'num_object_clones': 0, 'num_object_copies': 6164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10873, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -720}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10965", 'reported_seq': 11088, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:38.052563+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:38.052563+0000', 'last_peered': '2024-09-11T05:05:38.052563+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:38.052563+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12825760, 'num_objects': 1566, 'num_object_clones': 0, 'num_object_copies': 6264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10963, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -655}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:40.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11103", 'reported_seq': 11208, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:37.965148+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:37.965148+0000', 'last_peered': '2024-09-11T05:05:37.965148+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:37.965148+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12591104, 'num_objects': 1537, 'num_object_clones': 0, 'num_object_copies': 6148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11103, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -757}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11028", 'reported_seq': 11122, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:35.386993+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:35.386993+0000', 'last_peered': '2024-09-11T05:05:35.386993+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:35.386993+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13939872, 'num_objects': 1702, 'num_object_clones': 0, 'num_object_copies': 6808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11026, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -605}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11187", 'reported_seq': 11310, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:37.956562+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:37.956562+0000', 'last_peered': '2024-09-11T05:05:37.956562+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:37.956562+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13109568, 'num_objects': 1601, 'num_object_clones': 0, 'num_object_copies': 6404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11184, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -765}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10967", 'reported_seq': 11080, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:38.129384+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:38.129384+0000', 'last_peered': '2024-09-11T05:05:38.129384+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:38.129384+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12784800, 'num_objects': 1561, 'num_object_clones': 0, 'num_object_copies': 6244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10965, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -745}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11074", 'reported_seq': 11190, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:38.005604+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:38.005604+0000', 'last_peered': '2024-09-11T05:05:38.005604+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:38.005604+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12930800, 'num_objects': 1579, 'num_object_clones': 0, 'num_object_copies': 6316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11071, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -719}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:40.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11165", 'reported_seq': 14532, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:38.361160+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:38.361160+0000', 'last_peered': '2024-09-11T05:05:38.361160+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:38.361160+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12761680, 'num_objects': 1558, 'num_object_clones': 0, 'num_object_copies': 6232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 691, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11164, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1643, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 867}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:40.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:40.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11002", 'reported_seq': 11109, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:37.668679+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:37.668679+0000', 'last_peered': '2024-09-11T05:05:37.668679+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:37.668679+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13022368, 'num_objects': 1590, 'num_object_clones': 0, 'num_object_copies': 6360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11000, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -674}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:40.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:40.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10931", 'reported_seq': 11048, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:38.358984+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:38.358984+0000', 'last_peered': '2024-09-11T05:05:38.358984+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:38.358984+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12104864, 'num_objects': 1478, 'num_object_clones': 0, 'num_object_copies': 5912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10929, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:40.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:40.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11138", 'reported_seq': 11258, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:35.240820+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:35.240820+0000', 'last_peered': '2024-09-11T05:05:35.240820+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:35.240820+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13423776, 'num_objects': 1639, 'num_object_clones': 0, 'num_object_copies': 6556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11136, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:40.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:41.463 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:41.464 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30382/101152 objects degraded (30.036%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":668,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":25288,"data_bytes":207114728,"bytes_used":2887675904,"bytes_avail":383659380736,"bytes_total":386547056640,"degraded_objects":30382,"degraded_total":101152,"degraded_ratio":0.30035985447643149,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:41.486 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:41.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:42.163 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:42.164 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:42.182 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-11T05:05:41.872758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205148648,"num_objects":25048,"num_object_clones":0,"num_object_copies":100192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160855,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1926,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160759,"ondisk_log_size":160759,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2820372,"kb_used_data":403520,"kb_used_omap":14,"kb_used_meta":2416753,"kb_avail":374666988,"statfs":{"total":386547056640,"available":383658995712,"internally_reserved":0,"allocated":413204480,"data_stored":412447672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2474755638},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,3,7,11,5,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17981440,"num_objects":-2195,"num_object_clones":0,"num_object_copies":-8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004862"},"pg_stats":[{"pgid":"1.f","version":"35'10916","reported_seq":11011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.875334+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:37.875334+0000","last_peered":"2024-09-11T05:05:37.875334+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:37.875334+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13090816,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11135","reported_seq":11250,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:40.573981+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:40.573981+0000","last_peered":"2024-09-11T05:05:40.573981+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:40.573981+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13128864,"num_objects":1603,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11133,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10616","reported_seq":10721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.256583+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:38.256583+0000","last_peered":"2024-09-11T05:05:38.256583+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:38.256583+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12647016,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10615,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10931","reported_seq":11040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:40.272770+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:40.272770+0000","last_peered":"2024-09-11T05:05:40.272770+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:40.272770+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162208,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10965","reported_seq":11088,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.052563+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:38.052563+0000","last_peered":"2024-09-11T05:05:38.052563+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:38.052563+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12825760,"num_objects":1566,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11103","reported_seq":11208,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.965148+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:37.965148+0000","last_peered":"2024-09-11T05:05:37.965148+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:37.965148+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12591104,"num_objects":1537,"num_object_clones":0,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11103,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11096","reported_seq":11190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:40.404172+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:40.404172+0000","last_peered":"2024-09-11T05:05:40.404172+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:40.404172+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13382816,"num_objects":1634,"num_object_clones":0,"num_object_copies":6536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-673}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11234","reported_seq":16092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:38.598938+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:38.598938+0000","last_peered":"2024-09-11T05:05:38.598938+0000","last_clean":"2024-09-11T05:05:38.598938+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:38.598938+0000","last_undegraded":"2024-09-11T05:05:38.598938+0000","last_fullsized":"2024-09-11T05:05:38.598938+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13017088,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11187","reported_seq":11310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.956562+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:37.956562+0000","last_peered":"2024-09-11T05:05:37.956562+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:37.956562+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109568,"num_objects":1601,"num_object_clones":0,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11184,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10967","reported_seq":11080,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.129384+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:38.129384+0000","last_peered":"2024-09-11T05:05:38.129384+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:38.129384+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12784800,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11074","reported_seq":11190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.005604+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:38.005604+0000","last_peered":"2024-09-11T05:05:38.005604+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:38.005604+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12930800,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10840","reported_seq":15441,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:38.598915+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:38.598915+0000","last_peered":"2024-09-11T05:05:38.598915+0000","last_clean":"2024-09-11T05:05:38.598915+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:38.598915+0000","last_undegraded":"2024-09-11T05:05:38.598915+0000","last_fullsized":"2024-09-11T05:05:38.598915+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648448,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11165","reported_seq":14532,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.361160+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:38.361160+0000","last_peered":"2024-09-11T05:05:38.361160+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:38.361160+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12761680,"num_objects":1558,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11164,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":867}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11002","reported_seq":11109,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:37.668679+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:37.668679+0000","last_peered":"2024-09-11T05:05:37.668679+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:37.668679+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13022368,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11000,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-674}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10931","reported_seq":11048,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:38.358984+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:38.358984+0000","last_peered":"2024-09-11T05:05:38.358984+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:38.358984+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12104864,"num_objects":1478,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11197","reported_seq":11317,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:40.544343+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:40.544343+0000","last_peered":"2024-09-11T05:05:40.544343+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:40.544343+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12940448,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-708}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":205148648,"num_objects":25048,"num_object_clones":0,"num_object_copies":100192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160855,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1926,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":410058752,"data_stored":410058752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160759,"ondisk_log_size":160759,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750920,"kb_used_data":101432,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93620920,"statfs":{"total":96636764160,"available":95867822080,"internally_reserved":0,"allocated":103866368,"data_stored":103677166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,3,1,2,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728256,"kb_used_data":101476,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93643584,"statfs":{"total":96636764160,"available":95891030016,"internally_reserved":0,"allocated":103911424,"data_stored":103722222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614160,"kb_used_data":100792,"kb_used_omap":4,"kb_used_meta":513339,"kb_avail":93757680,"statfs":{"total":96636764160,"available":96007864320,"internally_reserved":0,"allocated":103211008,"data_stored":103021806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525659858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727036,"kb_used_data":99820,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93644804,"statfs":{"total":96636764160,"available":95892279296,"internally_reserved":0,"allocated":102215680,"data_stored":102026478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101236736,"data_stored":101236736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102703104,"data_stored":102703104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102948864,"data_stored":102948864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":103170048,"data_stored":103170048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:45.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:45.824 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:45.850 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-11T05:05:43.873591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":200282600,"num_objects":24454,"num_object_clones":0,"num_object_copies":97816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161449,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2821196,"kb_used_data":396032,"kb_used_omap":14,"kb_used_meta":2425073,"kb_avail":374666164,"statfs":{"total":386547056640,"available":383658151936,"internally_reserved":0,"allocated":405536768,"data_stored":404779960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2483275318},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,1,7,16,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20881408,"num_objects":-2549,"num_object_clones":0,"num_object_copies":-10196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2549,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004830"},"pg_stats":[{"pgid":"1.f","version":"35'10972","reported_seq":11067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.006205+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:43.006205+0000","last_peered":"2024-09-11T05:05:43.006205+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:43.006205+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12632064,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10972,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11135","reported_seq":11250,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:40.573981+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:40.573981+0000","last_peered":"2024-09-11T05:05:40.573981+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:40.573981+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13128864,"num_objects":1603,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11133,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10687","reported_seq":10791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.278844+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:43.278844+0000","last_peered":"2024-09-11T05:05:43.278844+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:43.278844+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12065384,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10686,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10931","reported_seq":11040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:40.272770+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:40.272770+0000","last_peered":"2024-09-11T05:05:40.272770+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:40.272770+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12162208,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11024","reported_seq":11148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.031630+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:43.031630+0000","last_peered":"2024-09-11T05:05:43.031630+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:43.031630+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12342432,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11149","reported_seq":11254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.143914+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:43.143914+0000","last_peered":"2024-09-11T05:05:43.143914+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:43.143914+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12214272,"num_objects":1491,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11096","reported_seq":11190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:40.404172+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:40.404172+0000","last_peered":"2024-09-11T05:05:40.404172+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:40.404172+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13382816,"num_objects":1634,"num_object_clones":0,"num_object_copies":6536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-673}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11234","reported_seq":16092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:38.598938+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:38.598938+0000","last_peered":"2024-09-11T05:05:38.598938+0000","last_clean":"2024-09-11T05:05:38.598938+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:38.598938+0000","last_undegraded":"2024-09-11T05:05:38.598938+0000","last_fullsized":"2024-09-11T05:05:38.598938+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13017088,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11248","reported_seq":11369,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:42.989443+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:42.989443+0000","last_peered":"2024-09-11T05:05:42.989443+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:42.989443+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12609856,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11245,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11042","reported_seq":11153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.349035+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:43.349035+0000","last_peered":"2024-09-11T05:05:43.349035+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:43.349035+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12170400,"num_objects":1486,"num_object_clones":0,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11040,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-820}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11134","reported_seq":11251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.008472+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:43.008472+0000","last_peered":"2024-09-11T05:05:43.008472+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:43.008472+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12439280,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11131,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10840","reported_seq":15441,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:38.598915+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:38.598915+0000","last_peered":"2024-09-11T05:05:38.598915+0000","last_clean":"2024-09-11T05:05:38.598915+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:38.598915+0000","last_undegraded":"2024-09-11T05:05:38.598915+0000","last_fullsized":"2024-09-11T05:05:38.598915+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648448,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11209","reported_seq":14711,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.418730+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:43.418730+0000","last_peered":"2024-09-11T05:05:43.418730+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:43.418730+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12401232,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11056","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:42.842715+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:42.842715+0000","last_peered":"2024-09-11T05:05:42.842715+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:42.842715+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12580000,"num_objects":1536,"num_object_clones":0,"num_object_copies":6144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-728}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10999","reported_seq":11116,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.401569+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:43.401569+0000","last_peered":"2024-09-11T05:05:43.401569+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:43.401569+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11547808,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11197","reported_seq":11317,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:40.544343+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:40.544343+0000","last_peered":"2024-09-11T05:05:40.544343+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:40.544343+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12940448,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-708}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":200282600,"num_objects":24454,"num_object_clones":0,"num_object_copies":97816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161449,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":402219008,"data_stored":402219008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751284,"kb_used_data":97636,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93620556,"statfs":{"total":96636764160,"available":95867449344,"internally_reserved":0,"allocated":99979264,"data_stored":99790062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728716,"kb_used_data":97784,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93643124,"statfs":{"total":96636764160,"available":95890558976,"internally_reserved":0,"allocated":100130816,"data_stored":99941614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,6,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614160,"kb_used_data":100792,"kb_used_omap":4,"kb_used_meta":513339,"kb_avail":93757680,"statfs":{"total":96636764160,"available":96007864320,"internally_reserved":0,"allocated":103211008,"data_stored":103021806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":525659858},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727036,"kb_used_data":99820,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93644804,"statfs":{"total":96636764160,"available":95892279296,"internally_reserved":0,"allocated":102215680,"data_stored":102026478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101236736,"data_stored":101236736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102703104,"data_stored":102703104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99033088,"data_stored":99033088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99246080,"data_stored":99246080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:45.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:46.522 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:46.523 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:46.541 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-11T05:05:45.874455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":197235176,"num_objects":24082,"num_object_clones":0,"num_object_copies":96328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24082,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161821,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822508,"kb_used_data":387616,"kb_used_omap":14,"kb_used_meta":2434801,"kb_avail":374664852,"statfs":{"total":386547056640,"available":383656808448,"internally_reserved":0,"allocated":396918784,"data_stored":396161976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2493236790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,7,13,15,14,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17842176,"num_objects":-2178,"num_object_clones":0,"num_object_copies":-8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2178,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004878"},"pg_stats":[{"pgid":"1.f","version":"35'10972","reported_seq":11067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.006205+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:43.006205+0000","last_peered":"2024-09-11T05:05:43.006205+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:43.006205+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12632064,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10972,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11197","reported_seq":11312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.468032+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:45.468032+0000","last_peered":"2024-09-11T05:05:45.468032+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:45.468032+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12620960,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10687","reported_seq":10791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.278844+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:43.278844+0000","last_peered":"2024-09-11T05:05:43.278844+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:43.278844+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12065384,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10686,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10984","reported_seq":11093,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.478624+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:45.478624+0000","last_peered":"2024-09-11T05:05:45.478624+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:45.478624+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11728032,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11024","reported_seq":11148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.031630+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:43.031630+0000","last_peered":"2024-09-11T05:05:43.031630+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:43.031630+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12342432,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11149","reported_seq":11254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.143914+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:43.143914+0000","last_peered":"2024-09-11T05:05:43.143914+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:43.143914+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12214272,"num_objects":1491,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11165","reported_seq":11259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.531050+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:45.531050+0000","last_peered":"2024-09-11T05:05:45.531050+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:45.531050+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12817568,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-742}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11304","reported_seq":16161,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:43.855785+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:43.855785+0000","last_peered":"2024-09-11T05:05:43.855785+0000","last_clean":"2024-09-11T05:05:43.855785+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:43.855785+0000","last_undegraded":"2024-09-11T05:05:43.855785+0000","last_fullsized":"2024-09-11T05:05:43.855785+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12443648,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2014,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11248","reported_seq":11369,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:42.989443+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:42.989443+0000","last_peered":"2024-09-11T05:05:42.989443+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:42.989443+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12609856,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11245,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11042","reported_seq":11153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.349035+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:43.349035+0000","last_peered":"2024-09-11T05:05:43.349035+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:43.349035+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12170400,"num_objects":1486,"num_object_clones":0,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11040,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-820}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11134","reported_seq":11251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.008472+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:43.008472+0000","last_peered":"2024-09-11T05:05:43.008472+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:43.008472+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12439280,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11131,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10888","reported_seq":15489,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:43.651299+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:43.651299+0000","last_peered":"2024-09-11T05:05:43.651299+0000","last_clean":"2024-09-11T05:05:43.651299+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:43.651299+0000","last_undegraded":"2024-09-11T05:05:43.651299+0000","last_fullsized":"2024-09-11T05:05:43.651299+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":1496,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4698,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11209","reported_seq":14711,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.418730+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:43.418730+0000","last_peered":"2024-09-11T05:05:43.418730+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:43.418730+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12401232,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11056","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:42.842715+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:42.842715+0000","last_peered":"2024-09-11T05:05:42.842715+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:42.842715+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12580000,"num_objects":1536,"num_object_clones":0,"num_object_copies":6144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-728}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10999","reported_seq":11116,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.401569+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:43.401569+0000","last_peered":"2024-09-11T05:05:43.401569+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:43.401569+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11547808,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11267","reported_seq":11387,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.335283+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:45.335283+0000","last_peered":"2024-09-11T05:05:45.335283+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:45.335283+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12367008,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11265,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-778}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":197235176,"num_objects":24082,"num_object_clones":0,"num_object_copies":96328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24082,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161821,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":394170368,"data_stored":394170368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751284,"kb_used_data":97636,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93620556,"statfs":{"total":96636764160,"available":95867449344,"internally_reserved":0,"allocated":99979264,"data_stored":99790062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728716,"kb_used_data":97784,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93643124,"statfs":{"total":96636764160,"available":95890558976,"internally_reserved":0,"allocated":100130816,"data_stored":99941614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,6,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597792,"kb_used_data":96904,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93774048,"statfs":{"total":96636764160,"available":96024625152,"internally_reserved":0,"allocated":99229696,"data_stored":99040494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512880338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744716,"kb_used_data":95292,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93627124,"statfs":{"total":96636764160,"available":95874174976,"internally_reserved":0,"allocated":97579008,"data_stored":97389806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97193984,"data_stored":97193984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98697216,"data_stored":98697216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99033088,"data_stored":99033088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99246080,"data_stored":99246080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:46.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10972", 'reported_seq': 11067, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:43.006205+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:43.006205+0000', 'last_peered': '2024-09-11T05:05:43.006205+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:43.006205+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12632064, 'num_objects': 1542, 'num_object_clones': 0, 'num_object_copies': 6168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10972, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11197", 'reported_seq': 11312, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:45.468032+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:45.468032+0000', 'last_peered': '2024-09-11T05:05:45.468032+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:45.468032+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12620960, 'num_objects': 1541, 'num_object_clones': 0, 'num_object_copies': 6164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11195, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10687", 'reported_seq': 10791, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:43.278844+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:43.278844+0000', 'last_peered': '2024-09-11T05:05:43.278844+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:43.278844+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12065384, 'num_objects': 1474, 'num_object_clones': 0, 'num_object_copies': 5896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1474, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10686, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -737}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10984", 'reported_seq': 11093, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:45.478624+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:45.478624+0000', 'last_peered': '2024-09-11T05:05:45.478624+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:45.478624+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11728032, 'num_objects': 1432, 'num_object_clones': 0, 'num_object_copies': 5728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10982, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:46.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11024", 'reported_seq': 11148, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:43.031630+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:43.031630+0000', 'last_peered': '2024-09-11T05:05:43.031630+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:43.031630+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12342432, 'num_objects': 1507, 'num_object_clones': 0, 'num_object_copies': 6028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11022, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -714}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11149", 'reported_seq': 11254, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:43.143914+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:43.143914+0000', 'last_peered': '2024-09-11T05:05:43.143914+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:43.143914+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12214272, 'num_objects': 1491, 'num_object_clones': 0, 'num_object_copies': 5964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11149, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -803}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11165", 'reported_seq': 11259, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:45.531050+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:45.531050+0000', 'last_peered': '2024-09-11T05:05:45.531050+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:45.531050+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12817568, 'num_objects': 1565, 'num_object_clones': 0, 'num_object_copies': 6260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11163, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -742}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11248", 'reported_seq': 11369, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:42.989443+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:42.989443+0000', 'last_peered': '2024-09-11T05:05:42.989443+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:42.989443+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12609856, 'num_objects': 1540, 'num_object_clones': 0, 'num_object_copies': 6160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11245, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -826}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11042", 'reported_seq': 11153, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:43.349035+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:43.349035+0000', 'last_peered': '2024-09-11T05:05:43.349035+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:43.349035+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12170400, 'num_objects': 1486, 'num_object_clones': 0, 'num_object_copies': 5944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11040, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -820}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:46.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11134", 'reported_seq': 11251, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:43.008472+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:43.008472+0000', 'last_peered': '2024-09-11T05:05:43.008472+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:43.008472+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12439280, 'num_objects': 1519, 'num_object_clones': 0, 'num_object_copies': 6076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11131, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -779}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:46.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:46.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11209", 'reported_seq': 14711, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:43.418730+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:43.418730+0000', 'last_peered': '2024-09-11T05:05:43.418730+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:43.418730+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12401232, 'num_objects': 1514, 'num_object_clones': 0, 'num_object_copies': 6056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 625, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11208, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1709, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 889}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:46.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:46.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11056", 'reported_seq': 11163, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:42.842715+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:42.842715+0000', 'last_peered': '2024-09-11T05:05:42.842715+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:42.842715+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12580000, 'num_objects': 1536, 'num_object_clones': 0, 'num_object_copies': 6144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11054, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -728}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:46.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:46.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10999", 'reported_seq': 11116, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:43.401569+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:43.401569+0000', 'last_peered': '2024-09-11T05:05:43.401569+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:43.401569+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11547808, 'num_objects': 1410, 'num_object_clones': 0, 'num_object_copies': 5640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10997, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:46.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:46.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11267", 'reported_seq': 11387, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:45.335283+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:45.335283+0000', 'last_peered': '2024-09-11T05:05:45.335283+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:45.335283+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12367008, 'num_objects': 1510, 'num_object_clones': 0, 'num_object_copies': 6040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11265, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -778}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:46.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:47.297 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:47.298 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30316/96328 objects degraded (31.472%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":674,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24082,"data_bytes":197235176,"bytes_used":2890248192,"bytes_avail":383656808448,"bytes_total":386547056640,"degraded_objects":30316,"degraded_total":96328,"degraded_ratio":0.3147163856822523,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:47.329 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:47.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:47.947 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:47.949 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:47.987 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-11T05:05:47.875291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":197235176,"num_objects":24082,"num_object_clones":0,"num_object_copies":96328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24082,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161821,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822508,"kb_used_data":387616,"kb_used_omap":14,"kb_used_meta":2434801,"kb_avail":374664852,"statfs":{"total":386547056640,"available":383656808448,"internally_reserved":0,"allocated":396918784,"data_stored":396161976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2493236790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,7,13,15,14,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15736832,"num_objects":-1921,"num_object_clones":0,"num_object_copies":-7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1921,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004840"},"pg_stats":[{"pgid":"1.f","version":"35'10972","reported_seq":11067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.006205+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:43.006205+0000","last_peered":"2024-09-11T05:05:43.006205+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:43.006205+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12632064,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10972,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11197","reported_seq":11312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.468032+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:45.468032+0000","last_peered":"2024-09-11T05:05:45.468032+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:45.468032+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12620960,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10687","reported_seq":10791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.278844+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:43.278844+0000","last_peered":"2024-09-11T05:05:43.278844+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:43.278844+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12065384,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10686,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10984","reported_seq":11093,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.478624+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:45.478624+0000","last_peered":"2024-09-11T05:05:45.478624+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:45.478624+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11728032,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11024","reported_seq":11148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.031630+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:43.031630+0000","last_peered":"2024-09-11T05:05:43.031630+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:43.031630+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12342432,"num_objects":1507,"num_object_clones":0,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11149","reported_seq":11254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.143914+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:43.143914+0000","last_peered":"2024-09-11T05:05:43.143914+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:43.143914+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12214272,"num_objects":1491,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11165","reported_seq":11259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.531050+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:45.531050+0000","last_peered":"2024-09-11T05:05:45.531050+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:45.531050+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12817568,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-742}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11304","reported_seq":16161,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:43.855785+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:43.855785+0000","last_peered":"2024-09-11T05:05:43.855785+0000","last_clean":"2024-09-11T05:05:43.855785+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:43.855785+0000","last_undegraded":"2024-09-11T05:05:43.855785+0000","last_fullsized":"2024-09-11T05:05:43.855785+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12443648,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2014,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11248","reported_seq":11369,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:42.989443+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:42.989443+0000","last_peered":"2024-09-11T05:05:42.989443+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:42.989443+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12609856,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11245,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11042","reported_seq":11153,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.349035+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:43.349035+0000","last_peered":"2024-09-11T05:05:43.349035+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:43.349035+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12170400,"num_objects":1486,"num_object_clones":0,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11040,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-820}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11134","reported_seq":11251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.008472+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:43.008472+0000","last_peered":"2024-09-11T05:05:43.008472+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:43.008472+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12439280,"num_objects":1519,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11131,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10888","reported_seq":15489,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:43.651299+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:43.651299+0000","last_peered":"2024-09-11T05:05:43.651299+0000","last_clean":"2024-09-11T05:05:43.651299+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:43.651299+0000","last_undegraded":"2024-09-11T05:05:43.651299+0000","last_fullsized":"2024-09-11T05:05:43.651299+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":1496,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4698,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11209","reported_seq":14711,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.418730+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:43.418730+0000","last_peered":"2024-09-11T05:05:43.418730+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:43.418730+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12401232,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1709,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11056","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:42.842715+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:42.842715+0000","last_peered":"2024-09-11T05:05:42.842715+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:42.842715+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12580000,"num_objects":1536,"num_object_clones":0,"num_object_copies":6144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-728}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10999","reported_seq":11116,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:43.401569+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:43.401569+0000","last_peered":"2024-09-11T05:05:43.401569+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:43.401569+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11547808,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11267","reported_seq":11387,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.335283+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:45.335283+0000","last_peered":"2024-09-11T05:05:45.335283+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:45.335283+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12367008,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11265,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-778}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":197235176,"num_objects":24082,"num_object_clones":0,"num_object_copies":96328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24082,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161821,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":394170368,"data_stored":394170368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751284,"kb_used_data":97636,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93620556,"statfs":{"total":96636764160,"available":95867449344,"internally_reserved":0,"allocated":99979264,"data_stored":99790062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728716,"kb_used_data":97784,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93643124,"statfs":{"total":96636764160,"available":95890558976,"internally_reserved":0,"allocated":100130816,"data_stored":99941614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,6,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597792,"kb_used_data":96904,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93774048,"statfs":{"total":96636764160,"available":96024625152,"internally_reserved":0,"allocated":99229696,"data_stored":99040494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512880338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744716,"kb_used_data":95292,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93627124,"statfs":{"total":96636764160,"available":95874174976,"internally_reserved":0,"allocated":97579008,"data_stored":97389806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97193984,"data_stored":97193984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98697216,"data_stored":98697216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99033088,"data_stored":99033088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99246080,"data_stored":99246080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:50.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:51.606 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:51.638 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-11T05:05:49.876154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":191369704,"num_objects":23366,"num_object_clones":0,"num_object_copies":93464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162537,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2842148,"kb_used_data":375328,"kb_used_omap":14,"kb_used_meta":2466737,"kb_avail":374645212,"statfs":{"total":386547056640,"available":383636697088,"internally_reserved":0,"allocated":384335872,"data_stored":383579064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2525939254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,19,8,14,12,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21602304,"num_objects":-2637,"num_object_clones":0,"num_object_copies":-10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2637,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005034"},"pg_stats":[{"pgid":"1.f","version":"35'11041","reported_seq":11136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.064154+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:48.064154+0000","last_peered":"2024-09-11T05:05:48.064154+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:48.064154+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12066816,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11197","reported_seq":11312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.468032+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:45.468032+0000","last_peered":"2024-09-11T05:05:45.468032+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:45.468032+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12620960,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10743","reported_seq":10848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.445125+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:48.445125+0000","last_peered":"2024-09-11T05:05:48.445125+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:48.445125+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11606632,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10742,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-793}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10984","reported_seq":11093,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.478624+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:45.478624+0000","last_peered":"2024-09-11T05:05:45.478624+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:45.478624+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11728032,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11080","reported_seq":11204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:47.921375+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:47.921375+0000","last_peered":"2024-09-11T05:05:47.921375+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:47.921375+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11883680,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11214","reported_seq":11320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.110620+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:48.110620+0000","last_peered":"2024-09-11T05:05:48.110620+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:48.110620+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11681792,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11165","reported_seq":11259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.531050+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:45.531050+0000","last_peered":"2024-09-11T05:05:45.531050+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:45.531050+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12817568,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-742}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11364","reported_seq":16222,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:48.777192+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:48.777192+0000","last_peered":"2024-09-11T05:05:48.777192+0000","last_clean":"2024-09-11T05:05:48.777192+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:48.777192+0000","last_undegraded":"2024-09-11T05:05:48.777192+0000","last_fullsized":"2024-09-11T05:05:48.777192+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11952128,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11313","reported_seq":11436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.198190+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:48.198190+0000","last_peered":"2024-09-11T05:05:48.198190+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:48.198190+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12077376,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11310,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11102","reported_seq":11214,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.321343+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:48.321343+0000","last_peered":"2024-09-11T05:05:48.321343+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:48.321343+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678880,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11197","reported_seq":11314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.260282+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:48.260282+0000","last_peered":"2024-09-11T05:05:48.260282+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:48.260282+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11923184,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11194,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10945","reported_seq":15546,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:48.653561+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:48.653561+0000","last_peered":"2024-09-11T05:05:48.653561+0000","last_clean":"2024-09-11T05:05:48.653561+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:48.653561+0000","last_undegraded":"2024-09-11T05:05:48.653561+0000","last_fullsized":"2024-09-11T05:05:48.653561+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11788288,"num_objects":1439,"num_object_clones":0,"num_object_copies":5756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11269","reported_seq":14887,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.445807+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:48.445807+0000","last_peered":"2024-09-11T05:05:48.445807+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:48.445807+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11909712,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11268,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1767,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11113","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.031226+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:48.031226+0000","last_peered":"2024-09-11T05:05:48.031226+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:48.031226+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12113056,"num_objects":1479,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-785}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11047","reported_seq":11166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.404029+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:48.404029+0000","last_peered":"2024-09-11T05:05:48.404029+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:48.404029+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11154592,"num_objects":1362,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-941}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11267","reported_seq":11387,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:45.335283+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:45.335283+0000","last_peered":"2024-09-11T05:05:45.335283+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:45.335283+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12367008,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11265,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-778}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":191369704,"num_objects":23366,"num_object_clones":0,"num_object_copies":93464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162537,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":382189568,"data_stored":382189568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753020,"kb_used_data":93804,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93618820,"statfs":{"total":96636764160,"available":95865671680,"internally_reserved":0,"allocated":96055296,"data_stored":95866094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,2,6,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746288,"kb_used_data":93152,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93625552,"statfs":{"total":96636764160,"available":95872565248,"internally_reserved":0,"allocated":95387648,"data_stored":95198446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598124,"kb_used_data":93080,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93773716,"statfs":{"total":96636764160,"available":96024285184,"internally_reserved":0,"allocated":95313920,"data_stored":95124718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517140178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,7,0,2,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744716,"kb_used_data":95292,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93627124,"statfs":{"total":96636764160,"available":95874174976,"internally_reserved":0,"allocated":97579008,"data_stored":97389806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97193984,"data_stored":97193984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94707712,"data_stored":94707712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95064064,"data_stored":95064064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95223808,"data_stored":95223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:51.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:52.293 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:52.293 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:52.317 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-11T05:05:51.877029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189288936,"num_objects":23112,"num_object_clones":0,"num_object_copies":92448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23112,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162791,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2842476,"kb_used_data":371484,"kb_used_omap":14,"kb_used_meta":2470897,"kb_avail":374644884,"statfs":{"total":386547056640,"available":383636361216,"internally_reserved":0,"allocated":380399616,"data_stored":379642808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2530199094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,16,9,11,11,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17825792,"num_objects":-2176,"num_object_clones":0,"num_object_copies":-8704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2176,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005184"},"pg_stats":[{"pgid":"1.f","version":"35'11041","reported_seq":11136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.064154+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:48.064154+0000","last_peered":"2024-09-11T05:05:48.064154+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:48.064154+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12066816,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11256","reported_seq":11371,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.372503+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:50.372503+0000","last_peered":"2024-09-11T05:05:50.372503+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:50.372503+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12137632,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10743","reported_seq":10848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.445125+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:48.445125+0000","last_peered":"2024-09-11T05:05:48.445125+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:48.445125+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11606632,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10742,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-793}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11045","reported_seq":11154,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.396800+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:50.396800+0000","last_peered":"2024-09-11T05:05:50.396800+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:50.396800+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11228320,"num_objects":1371,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11080","reported_seq":11204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:47.921375+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:47.921375+0000","last_peered":"2024-09-11T05:05:47.921375+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:47.921375+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11883680,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11214","reported_seq":11320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.110620+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:48.110620+0000","last_peered":"2024-09-11T05:05:48.110620+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:48.110620+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11681792,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11243","reported_seq":11337,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.789641+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:50.789641+0000","last_peered":"2024-09-11T05:05:50.789641+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:50.789641+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178592,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11364","reported_seq":16222,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:48.777192+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:48.777192+0000","last_peered":"2024-09-11T05:05:48.777192+0000","last_clean":"2024-09-11T05:05:48.777192+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:48.777192+0000","last_undegraded":"2024-09-11T05:05:48.777192+0000","last_fullsized":"2024-09-11T05:05:48.777192+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11952128,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11313","reported_seq":11436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.198190+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:48.198190+0000","last_peered":"2024-09-11T05:05:48.198190+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:48.198190+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12077376,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11310,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11102","reported_seq":11214,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.321343+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:48.321343+0000","last_peered":"2024-09-11T05:05:48.321343+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:48.321343+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678880,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11197","reported_seq":11314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.260282+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:48.260282+0000","last_peered":"2024-09-11T05:05:48.260282+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:48.260282+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11923184,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11194,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10945","reported_seq":15546,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:48.653561+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:48.653561+0000","last_peered":"2024-09-11T05:05:48.653561+0000","last_clean":"2024-09-11T05:05:48.653561+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:48.653561+0000","last_undegraded":"2024-09-11T05:05:48.653561+0000","last_fullsized":"2024-09-11T05:05:48.653561+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11788288,"num_objects":1439,"num_object_clones":0,"num_object_copies":5756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11269","reported_seq":14887,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.445807+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:48.445807+0000","last_peered":"2024-09-11T05:05:48.445807+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:48.445807+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11909712,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11268,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1767,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11113","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.031226+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:48.031226+0000","last_peered":"2024-09-11T05:05:48.031226+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:48.031226+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12113056,"num_objects":1479,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-785}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11047","reported_seq":11166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.404029+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:48.404029+0000","last_peered":"2024-09-11T05:05:48.404029+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:48.404029+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11154592,"num_objects":1362,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-941}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11323","reported_seq":11443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.660236+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:50.660236+0000","last_peered":"2024-09-11T05:05:50.660236+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:50.660236+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11908256,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11321,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189288936,"num_objects":23112,"num_object_clones":0,"num_object_copies":92448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23112,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162791,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378122240,"data_stored":378122240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753020,"kb_used_data":93804,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93618820,"statfs":{"total":96636764160,"available":95865671680,"internally_reserved":0,"allocated":96055296,"data_stored":95866094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,2,6,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746288,"kb_used_data":93152,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93625552,"statfs":{"total":96636764160,"available":95872565248,"internally_reserved":0,"allocated":95387648,"data_stored":95198446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598124,"kb_used_data":93080,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93773716,"statfs":{"total":96636764160,"available":96024285184,"internally_reserved":0,"allocated":95313920,"data_stored":95124718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517140178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,7,0,2,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745044,"kb_used_data":91448,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93626796,"statfs":{"total":96636764160,"available":95873839104,"internally_reserved":0,"allocated":93642752,"data_stored":93453550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93126656,"data_stored":93126656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94707712,"data_stored":94707712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95064064,"data_stored":95064064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95223808,"data_stored":95223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:52.318 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:52.318 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11041", 'reported_seq': 11136, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.064154+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:48.064154+0000', 'last_peered': '2024-09-11T05:05:48.064154+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:48.064154+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12066816, 'num_objects': 1473, 'num_object_clones': 0, 'num_object_copies': 5892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11041, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -818}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:52.318 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:52.318 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11256", 'reported_seq': 11371, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:50.372503+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:50.372503+0000', 'last_peered': '2024-09-11T05:05:50.372503+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:50.372503+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12137632, 'num_objects': 1482, 'num_object_clones': 0, 'num_object_copies': 5928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11254, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:52.318 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:52.318 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10743", 'reported_seq': 10848, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.445125+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:48.445125+0000', 'last_peered': '2024-09-11T05:05:48.445125+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:48.445125+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11606632, 'num_objects': 1418, 'num_object_clones': 0, 'num_object_copies': 5672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1418, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10742, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -793}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11045", 'reported_seq': 11154, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:50.396800+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:50.396800+0000', 'last_peered': '2024-09-11T05:05:50.396800+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:50.396800+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11228320, 'num_objects': 1371, 'num_object_clones': 0, 'num_object_copies': 5484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11043, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11080", 'reported_seq': 11204, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:47.921375+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:47.921375+0000', 'last_peered': '2024-09-11T05:05:47.921375+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:47.921375+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11883680, 'num_objects': 1451, 'num_object_clones': 0, 'num_object_copies': 5804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11078, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11214", 'reported_seq': 11320, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.110620+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:48.110620+0000', 'last_peered': '2024-09-11T05:05:48.110620+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:48.110620+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11681792, 'num_objects': 1426, 'num_object_clones': 0, 'num_object_copies': 5704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1426, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11214, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -868}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11243", 'reported_seq': 11337, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:50.789641+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:50.789641+0000', 'last_peered': '2024-09-11T05:05:50.789641+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:50.789641+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12178592, 'num_objects': 1487, 'num_object_clones': 0, 'num_object_copies': 5948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11241, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -820}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:52.319 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11313", 'reported_seq': 11436, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.198190+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:48.198190+0000', 'last_peered': '2024-09-11T05:05:48.198190+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:48.198190+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12077376, 'num_objects': 1475, 'num_object_clones': 0, 'num_object_copies': 5900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11310, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -891}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11102", 'reported_seq': 11214, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.321343+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:48.321343+0000', 'last_peered': '2024-09-11T05:05:48.321343+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:48.321343+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11678880, 'num_objects': 1426, 'num_object_clones': 0, 'num_object_copies': 5704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1426, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11100, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -880}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11197", 'reported_seq': 11314, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.260282+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:48.260282+0000', 'last_peered': '2024-09-11T05:05:48.260282+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:48.260282+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11923184, 'num_objects': 1456, 'num_object_clones': 0, 'num_object_copies': 5824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11194, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -842}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11269", 'reported_seq': 14887, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.445807+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:48.445807+0000', 'last_peered': '2024-09-11T05:05:48.445807+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:48.445807+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11909712, 'num_objects': 1454, 'num_object_clones': 0, 'num_object_copies': 5816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 567, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11268, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1767, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11113", 'reported_seq': 11220, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.031226+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:48.031226+0000', 'last_peered': '2024-09-11T05:05:48.031226+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:48.031226+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12113056, 'num_objects': 1479, 'num_object_clones': 0, 'num_object_copies': 5916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11111, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -785}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:52.320 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11047", 'reported_seq': 11166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:48.404029+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:48.404029+0000', 'last_peered': '2024-09-11T05:05:48.404029+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:48.404029+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11154592, 'num_objects': 1362, 'num_object_clones': 0, 'num_object_copies': 5448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11045, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -941}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:52.321 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:52.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11323", 'reported_seq': 11443, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:50.660236+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:50.660236+0000', 'last_peered': '2024-09-11T05:05:50.660236+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:50.660236+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11908256, 'num_objects': 1454, 'num_object_clones': 0, 'num_object_copies': 5816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11321, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -834}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:52.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:53.113 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:53.113 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30258/92448 objects degraded (32.730%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":680,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23112,"data_bytes":189288936,"bytes_used":2910695424,"bytes_avail":383636361216,"bytes_total":386547056640,"degraded_objects":30258,"degraded_total":92448,"degraded_ratio":0.32729750778816197,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:53.134 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:53.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:53.815 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:53.846 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-11T05:05:51.877029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189288936,"num_objects":23112,"num_object_clones":0,"num_object_copies":92448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23112,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162791,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2842476,"kb_used_data":371484,"kb_used_omap":14,"kb_used_meta":2470897,"kb_avail":374644884,"statfs":{"total":386547056640,"available":383636361216,"internally_reserved":0,"allocated":380399616,"data_stored":379642808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2530199094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,16,9,11,11,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17825792,"num_objects":-2176,"num_object_clones":0,"num_object_copies":-8704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2176,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005184"},"pg_stats":[{"pgid":"1.f","version":"35'11041","reported_seq":11136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.064154+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:48.064154+0000","last_peered":"2024-09-11T05:05:48.064154+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:48.064154+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12066816,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11256","reported_seq":11371,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.372503+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:50.372503+0000","last_peered":"2024-09-11T05:05:50.372503+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:50.372503+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12137632,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10743","reported_seq":10848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.445125+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:48.445125+0000","last_peered":"2024-09-11T05:05:48.445125+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:48.445125+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11606632,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10742,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-793}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11045","reported_seq":11154,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.396800+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:50.396800+0000","last_peered":"2024-09-11T05:05:50.396800+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:50.396800+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11228320,"num_objects":1371,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11080","reported_seq":11204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:47.921375+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:47.921375+0000","last_peered":"2024-09-11T05:05:47.921375+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:47.921375+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11883680,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11214","reported_seq":11320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.110620+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:48.110620+0000","last_peered":"2024-09-11T05:05:48.110620+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:48.110620+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11681792,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11243","reported_seq":11337,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.789641+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:50.789641+0000","last_peered":"2024-09-11T05:05:50.789641+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:50.789641+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178592,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11364","reported_seq":16222,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:48.777192+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:48.777192+0000","last_peered":"2024-09-11T05:05:48.777192+0000","last_clean":"2024-09-11T05:05:48.777192+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:48.777192+0000","last_undegraded":"2024-09-11T05:05:48.777192+0000","last_fullsized":"2024-09-11T05:05:48.777192+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11952128,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11313","reported_seq":11436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.198190+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:48.198190+0000","last_peered":"2024-09-11T05:05:48.198190+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:48.198190+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12077376,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11310,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11102","reported_seq":11214,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.321343+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:48.321343+0000","last_peered":"2024-09-11T05:05:48.321343+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:48.321343+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678880,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11197","reported_seq":11314,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.260282+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:48.260282+0000","last_peered":"2024-09-11T05:05:48.260282+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:48.260282+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11923184,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11194,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10945","reported_seq":15546,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:48.653561+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:48.653561+0000","last_peered":"2024-09-11T05:05:48.653561+0000","last_clean":"2024-09-11T05:05:48.653561+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:48.653561+0000","last_undegraded":"2024-09-11T05:05:48.653561+0000","last_fullsized":"2024-09-11T05:05:48.653561+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11788288,"num_objects":1439,"num_object_clones":0,"num_object_copies":5756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11269","reported_seq":14887,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.445807+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:48.445807+0000","last_peered":"2024-09-11T05:05:48.445807+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:48.445807+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11909712,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11268,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1767,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11113","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.031226+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:48.031226+0000","last_peered":"2024-09-11T05:05:48.031226+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:48.031226+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12113056,"num_objects":1479,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-785}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11047","reported_seq":11166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:48.404029+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:48.404029+0000","last_peered":"2024-09-11T05:05:48.404029+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:48.404029+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11154592,"num_objects":1362,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11045,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-941}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11323","reported_seq":11443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.660236+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:50.660236+0000","last_peered":"2024-09-11T05:05:50.660236+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:50.660236+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11908256,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11321,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189288936,"num_objects":23112,"num_object_clones":0,"num_object_copies":92448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23112,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162791,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378122240,"data_stored":378122240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753020,"kb_used_data":93804,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93618820,"statfs":{"total":96636764160,"available":95865671680,"internally_reserved":0,"allocated":96055296,"data_stored":95866094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,2,6,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746288,"kb_used_data":93152,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93625552,"statfs":{"total":96636764160,"available":95872565248,"internally_reserved":0,"allocated":95387648,"data_stored":95198446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598124,"kb_used_data":93080,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93773716,"statfs":{"total":96636764160,"available":96024285184,"internally_reserved":0,"allocated":95313920,"data_stored":95124718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517140178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,7,0,2,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745044,"kb_used_data":91448,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93626796,"statfs":{"total":96636764160,"available":95873839104,"internally_reserved":0,"allocated":93642752,"data_stored":93453550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93126656,"data_stored":93126656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94707712,"data_stored":94707712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95064064,"data_stored":95064064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95223808,"data_stored":95223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:56.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:57.464 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:57.465 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:57.487 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-11T05:05:55.878172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183325160,"num_objects":22384,"num_object_clones":0,"num_object_copies":89536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163519,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2823268,"kb_used_data":360576,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374664092,"statfs":{"total":386547056640,"available":383656030208,"internally_reserved":0,"allocated":369229824,"data_stored":368473016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2521679414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,6,3,11,16,8,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16957440,"num_objects":-2070,"num_object_clones":0,"num_object_copies":-8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2070,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004581"},"pg_stats":[{"pgid":"1.f","version":"35'11102","reported_seq":11197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.378520+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:53.378520+0000","last_peered":"2024-09-11T05:05:53.378520+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:53.378520+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11567104,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11102,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11256","reported_seq":11371,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.372503+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:50.372503+0000","last_peered":"2024-09-11T05:05:50.372503+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:50.372503+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12137632,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10803","reported_seq":10908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:52.953632+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:52.953632+0000","last_peered":"2024-09-11T05:05:52.953632+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:52.953632+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11115112,"num_objects":1358,"num_object_clones":0,"num_object_copies":5432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10802,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-853}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11045","reported_seq":11154,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.396800+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:50.396800+0000","last_peered":"2024-09-11T05:05:50.396800+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:50.396800+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11228320,"num_objects":1371,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11132","reported_seq":11255,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.232480+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:53.232480+0000","last_peered":"2024-09-11T05:05:53.232480+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:53.232480+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11457696,"num_objects":1399,"num_object_clones":0,"num_object_copies":5596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11130,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11275","reported_seq":11381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.317122+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:53.317122+0000","last_peered":"2024-09-11T05:05:53.317122+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:53.317122+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182080,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11275,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11243","reported_seq":11337,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.789641+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:50.789641+0000","last_peered":"2024-09-11T05:05:50.789641+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:50.789641+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178592,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11430","reported_seq":16288,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:53.906373+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:53.906373+0000","last_peered":"2024-09-11T05:05:53.906373+0000","last_clean":"2024-09-11T05:05:53.906373+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:53.906373+0000","last_undegraded":"2024-09-11T05:05:53.906373+0000","last_fullsized":"2024-09-11T05:05:53.906373+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11411456,"num_objects":1393,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11371","reported_seq":11494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.367492+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:53.367492+0000","last_peered":"2024-09-11T05:05:53.367492+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:53.367492+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11602240,"num_objects":1417,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11186","reported_seq":11299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.407951+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:53.407951+0000","last_peered":"2024-09-11T05:05:53.407951+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:53.407951+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10990752,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11184,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11261","reported_seq":11378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.242967+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:53.242967+0000","last_peered":"2024-09-11T05:05:53.242967+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:53.242967+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11398896,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11258,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11004","reported_seq":15605,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:54.009765+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:54.009765+0000","last_peered":"2024-09-11T05:05:54.009765+0000","last_clean":"2024-09-11T05:05:54.009765+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:54.009765+0000","last_undegraded":"2024-09-11T05:05:54.009765+0000","last_fullsized":"2024-09-11T05:05:54.009765+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11304960,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4814,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11314","reported_seq":15055,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.503244+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:53.503244+0000","last_peered":"2024-09-11T05:05:53.503244+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:53.503244+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11541072,"num_objects":1409,"num_object_clones":0,"num_object_copies":5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11313,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":904}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11185","reported_seq":11292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.225394+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:53.225394+0000","last_peered":"2024-09-11T05:05:53.225394+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:53.225394+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11523232,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11183,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-857}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11093","reported_seq":11212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.177362+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:53.177362+0000","last_peered":"2024-09-11T05:05:53.177362+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:53.177362+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10777760,"num_objects":1316,"num_object_clones":0,"num_object_copies":5264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11091,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-987}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11323","reported_seq":11443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:50.660236+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:50.660236+0000","last_peered":"2024-09-11T05:05:50.660236+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:50.660236+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11908256,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11321,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":183325160,"num_objects":22384,"num_object_clones":0,"num_object_copies":89536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163519,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":366108672,"data_stored":366108672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753304,"kb_used_data":89928,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93618536,"statfs":{"total":96636764160,"available":95865380864,"internally_reserved":0,"allocated":92086272,"data_stored":91897070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,9,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730620,"kb_used_data":89964,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93641220,"statfs":{"total":96636764160,"available":95888609280,"internally_reserved":0,"allocated":92123136,"data_stored":91933934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594300,"kb_used_data":89236,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93777540,"statfs":{"total":96636764160,"available":96028200960,"internally_reserved":0,"allocated":91377664,"data_stored":91188462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517140178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745044,"kb_used_data":91448,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93626796,"statfs":{"total":96636764160,"available":95873839104,"internally_reserved":0,"allocated":93642752,"data_stored":93453550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93126656,"data_stored":93126656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90771456,"data_stored":90771456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91099136,"data_stored":91099136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91111424,"data_stored":91111424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:57.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:58.191 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:58.226 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-11T05:05:57.878837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":181162472,"num_objects":22120,"num_object_clones":0,"num_object_copies":88480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163783,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160687,"ondisk_log_size":160687,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806992,"kb_used_data":356788,"kb_used_omap":14,"kb_used_meta":2450097,"kb_avail":374680368,"statfs":{"total":386547056640,"available":383672696832,"internally_reserved":0,"allocated":365350912,"data_stored":364594104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2508899894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,1,7,18,5,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16072704,"num_objects":-1962,"num_object_clones":0,"num_object_copies":-7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004381"},"pg_stats":[{"pgid":"1.f","version":"35'11102","reported_seq":11197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.378520+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:53.378520+0000","last_peered":"2024-09-11T05:05:53.378520+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:53.378520+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11567104,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11102,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11318","reported_seq":11433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:55.616735+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:55.616735+0000","last_peered":"2024-09-11T05:05:55.616735+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:55.616735+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11629728,"num_objects":1420,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10803","reported_seq":10908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:52.953632+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:52.953632+0000","last_peered":"2024-09-11T05:05:52.953632+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:52.953632+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11115112,"num_objects":1358,"num_object_clones":0,"num_object_copies":5432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10802,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-853}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11111","reported_seq":11218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:55.711616+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:55.711616+0000","last_peered":"2024-09-11T05:05:55.711616+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:55.711616+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10687648,"num_objects":1305,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11109,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11132","reported_seq":11255,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.232480+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:53.232480+0000","last_peered":"2024-09-11T05:05:53.232480+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:53.232480+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11457696,"num_objects":1399,"num_object_clones":0,"num_object_copies":5596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11130,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11275","reported_seq":11381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.317122+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:53.317122+0000","last_peered":"2024-09-11T05:05:53.317122+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:53.317122+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182080,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11275,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11317","reported_seq":11412,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:55.851567+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:55.851567+0000","last_peered":"2024-09-11T05:05:55.851567+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:55.851567+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11572384,"num_objects":1413,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11315,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11430","reported_seq":16288,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:53.906373+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:53.906373+0000","last_peered":"2024-09-11T05:05:53.906373+0000","last_clean":"2024-09-11T05:05:53.906373+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:53.906373+0000","last_undegraded":"2024-09-11T05:05:53.906373+0000","last_fullsized":"2024-09-11T05:05:53.906373+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11411456,"num_objects":1393,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11371","reported_seq":11494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.367492+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:53.367492+0000","last_peered":"2024-09-11T05:05:53.367492+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:53.367492+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11602240,"num_objects":1417,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11186","reported_seq":11299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.407951+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:53.407951+0000","last_peered":"2024-09-11T05:05:53.407951+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:53.407951+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10990752,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11184,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11261","reported_seq":11378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.242967+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:53.242967+0000","last_peered":"2024-09-11T05:05:53.242967+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:53.242967+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11398896,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11258,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11004","reported_seq":15605,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:54.009765+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:54.009765+0000","last_peered":"2024-09-11T05:05:54.009765+0000","last_clean":"2024-09-11T05:05:54.009765+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:54.009765+0000","last_undegraded":"2024-09-11T05:05:54.009765+0000","last_fullsized":"2024-09-11T05:05:54.009765+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11304960,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4814,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11314","reported_seq":15055,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.503244+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:53.503244+0000","last_peered":"2024-09-11T05:05:53.503244+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:53.503244+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11541072,"num_objects":1409,"num_object_clones":0,"num_object_copies":5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11313,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":904}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11185","reported_seq":11292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.225394+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:53.225394+0000","last_peered":"2024-09-11T05:05:53.225394+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:53.225394+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11523232,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11183,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-857}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11093","reported_seq":11212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.177362+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:53.177362+0000","last_peered":"2024-09-11T05:05:53.177362+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:53.177362+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10777760,"num_objects":1316,"num_object_clones":0,"num_object_copies":5264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11091,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-987}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11385","reported_seq":11505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:55.814706+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:55.814706+0000","last_peered":"2024-09-11T05:05:55.814706+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:55.814706+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11400352,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11383,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":181162472,"num_objects":22120,"num_object_clones":0,"num_object_copies":88480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163783,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362123264,"data_stored":362123264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160687,"ondisk_log_size":160687,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753304,"kb_used_data":89928,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93618536,"statfs":{"total":96636764160,"available":95865380864,"internally_reserved":0,"allocated":92086272,"data_stored":91897070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,9,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730620,"kb_used_data":89964,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93641220,"statfs":{"total":96636764160,"available":95888609280,"internally_reserved":0,"allocated":92123136,"data_stored":91933934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594300,"kb_used_data":89236,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93777540,"statfs":{"total":96636764160,"available":96028200960,"internally_reserved":0,"allocated":91377664,"data_stored":91188462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517140178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728768,"kb_used_data":87660,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93643072,"statfs":{"total":96636764160,"available":95890505728,"internally_reserved":0,"allocated":89763840,"data_stored":89574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656469711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89141248,"data_stored":89141248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90771456,"data_stored":90771456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91099136,"data_stored":91099136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91111424,"data_stored":91111424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:05:58.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:05:58.227 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11102", 'reported_seq': 11197, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.378520+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:05:53.378520+0000', 'last_peered': '2024-09-11T05:05:53.378520+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:05:53.378520+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11567104, 'num_objects': 1412, 'num_object_clones': 0, 'num_object_copies': 5648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11102, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -879}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:58.227 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:05:58.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11318", 'reported_seq': 11433, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:55.616735+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:05:55.616735+0000', 'last_peered': '2024-09-11T05:05:55.616735+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:05:55.616735+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11629728, 'num_objects': 1420, 'num_object_clones': 0, 'num_object_copies': 5680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11316, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -838}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:58.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:05:58.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10803", 'reported_seq': 10908, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:52.953632+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:05:52.953632+0000', 'last_peered': '2024-09-11T05:05:52.953632+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:05:52.953632+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11115112, 'num_objects': 1358, 'num_object_clones': 0, 'num_object_copies': 5432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1358, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10802, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -853}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:58.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:05:58.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11111", 'reported_seq': 11218, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:55.711616+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:05:55.711616+0000', 'last_peered': '2024-09-11T05:05:55.711616+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:05:55.711616+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10687648, 'num_objects': 1305, 'num_object_clones': 0, 'num_object_copies': 5220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11109, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:58.228 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:05:58.228 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11132", 'reported_seq': 11255, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.232480+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:05:53.232480+0000', 'last_peered': '2024-09-11T05:05:53.232480+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:05:53.232480+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11457696, 'num_objects': 1399, 'num_object_clones': 0, 'num_object_copies': 5596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11130, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -822}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11275", 'reported_seq': 11381, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.317122+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:05:53.317122+0000', 'last_peered': '2024-09-11T05:05:53.317122+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:05:53.317122+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11182080, 'num_objects': 1365, 'num_object_clones': 0, 'num_object_copies': 5460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11275, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -929}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11317", 'reported_seq': 11412, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:55.851567+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:05:55.851567+0000', 'last_peered': '2024-09-11T05:05:55.851567+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:05:55.851567+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11572384, 'num_objects': 1413, 'num_object_clones': 0, 'num_object_copies': 5652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11315, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -894}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11371", 'reported_seq': 11494, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.367492+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:05:53.367492+0000', 'last_peered': '2024-09-11T05:05:53.367492+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:05:53.367492+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11602240, 'num_objects': 1417, 'num_object_clones': 0, 'num_object_copies': 5668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11368, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -949}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11186", 'reported_seq': 11299, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.407951+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:05:53.407951+0000', 'last_peered': '2024-09-11T05:05:53.407951+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:05:53.407951+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10990752, 'num_objects': 1342, 'num_object_clones': 0, 'num_object_copies': 5368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11184, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -964}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:05:58.229 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11261", 'reported_seq': 11378, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.242967+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:05:53.242967+0000', 'last_peered': '2024-09-11T05:05:53.242967+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:05:53.242967+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11398896, 'num_objects': 1392, 'num_object_clones': 0, 'num_object_copies': 5568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11258, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -906}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:58.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:05:58.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11314", 'reported_seq': 15055, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.503244+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:05:53.503244+0000', 'last_peered': '2024-09-11T05:05:53.503244+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:05:53.503244+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11541072, 'num_objects': 1409, 'num_object_clones': 0, 'num_object_copies': 5636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 505, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11313, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1829, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 904}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:58.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:05:58.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11185", 'reported_seq': 11292, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.225394+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:05:53.225394+0000', 'last_peered': '2024-09-11T05:05:53.225394+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:05:53.225394+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11523232, 'num_objects': 1407, 'num_object_clones': 0, 'num_object_copies': 5628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11183, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -857}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:05:58.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:05:58.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11093", 'reported_seq': 11212, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:53.177362+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:05:53.177362+0000', 'last_peered': '2024-09-11T05:05:53.177362+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:05:53.177362+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10777760, 'num_objects': 1316, 'num_object_clones': 0, 'num_object_copies': 5264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11091, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -987}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:05:58.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:05:58.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11385", 'reported_seq': 11505, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:05:55.814706+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:05:55.814706+0000', 'last_peered': '2024-09-11T05:05:55.814706+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:05:55.814706+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11400352, 'num_objects': 1392, 'num_object_clones': 0, 'num_object_copies': 5568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11383, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -896}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:05:58.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:05:59.016 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:59.016 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30196/88480 objects degraded (34.127%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":686,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22120,"data_bytes":181162472,"bytes_used":2874359808,"bytes_avail":383672696832,"bytes_total":386547056640,"degraded_objects":30196,"degraded_total":88480,"degraded_ratio":0.34127486437613025,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":163},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:05:59.040 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:05:59.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:05:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:05:59.709 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:05:59.736 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-11T05:05:57.878837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":181162472,"num_objects":22120,"num_object_clones":0,"num_object_copies":88480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163783,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160687,"ondisk_log_size":160687,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806992,"kb_used_data":356788,"kb_used_omap":14,"kb_used_meta":2450097,"kb_avail":374680368,"statfs":{"total":386547056640,"available":383672696832,"internally_reserved":0,"allocated":365350912,"data_stored":364594104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2508899894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,1,7,18,5,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16072704,"num_objects":-1962,"num_object_clones":0,"num_object_copies":-7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004381"},"pg_stats":[{"pgid":"1.f","version":"35'11102","reported_seq":11197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.378520+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:53.378520+0000","last_peered":"2024-09-11T05:05:53.378520+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:53.378520+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11567104,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11102,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11318","reported_seq":11433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:55.616735+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:05:55.616735+0000","last_peered":"2024-09-11T05:05:55.616735+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:05:55.616735+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11629728,"num_objects":1420,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10803","reported_seq":10908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:52.953632+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:52.953632+0000","last_peered":"2024-09-11T05:05:52.953632+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:52.953632+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11115112,"num_objects":1358,"num_object_clones":0,"num_object_copies":5432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10802,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-853}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11111","reported_seq":11218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:55.711616+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:05:55.711616+0000","last_peered":"2024-09-11T05:05:55.711616+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:05:55.711616+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10687648,"num_objects":1305,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11109,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11132","reported_seq":11255,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.232480+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:53.232480+0000","last_peered":"2024-09-11T05:05:53.232480+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:53.232480+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11457696,"num_objects":1399,"num_object_clones":0,"num_object_copies":5596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11130,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11275","reported_seq":11381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.317122+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:53.317122+0000","last_peered":"2024-09-11T05:05:53.317122+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:53.317122+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182080,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11275,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11317","reported_seq":11412,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:55.851567+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:05:55.851567+0000","last_peered":"2024-09-11T05:05:55.851567+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:05:55.851567+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11572384,"num_objects":1413,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11315,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11430","reported_seq":16288,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:53.906373+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:53.906373+0000","last_peered":"2024-09-11T05:05:53.906373+0000","last_clean":"2024-09-11T05:05:53.906373+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:53.906373+0000","last_undegraded":"2024-09-11T05:05:53.906373+0000","last_fullsized":"2024-09-11T05:05:53.906373+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11411456,"num_objects":1393,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11371","reported_seq":11494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.367492+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:53.367492+0000","last_peered":"2024-09-11T05:05:53.367492+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:53.367492+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11602240,"num_objects":1417,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11186","reported_seq":11299,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.407951+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:53.407951+0000","last_peered":"2024-09-11T05:05:53.407951+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:53.407951+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10990752,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11184,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11261","reported_seq":11378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.242967+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:53.242967+0000","last_peered":"2024-09-11T05:05:53.242967+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:53.242967+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11398896,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11258,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11004","reported_seq":15605,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:54.009765+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:54.009765+0000","last_peered":"2024-09-11T05:05:54.009765+0000","last_clean":"2024-09-11T05:05:54.009765+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:54.009765+0000","last_undegraded":"2024-09-11T05:05:54.009765+0000","last_fullsized":"2024-09-11T05:05:54.009765+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11304960,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4814,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11314","reported_seq":15055,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.503244+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:53.503244+0000","last_peered":"2024-09-11T05:05:53.503244+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:53.503244+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11541072,"num_objects":1409,"num_object_clones":0,"num_object_copies":5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11313,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":904}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11185","reported_seq":11292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.225394+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:53.225394+0000","last_peered":"2024-09-11T05:05:53.225394+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:53.225394+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11523232,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11183,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-857}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11093","reported_seq":11212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:53.177362+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:53.177362+0000","last_peered":"2024-09-11T05:05:53.177362+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:53.177362+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10777760,"num_objects":1316,"num_object_clones":0,"num_object_copies":5264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11091,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-987}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11385","reported_seq":11505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:55.814706+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:05:55.814706+0000","last_peered":"2024-09-11T05:05:55.814706+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:05:55.814706+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11400352,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11383,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":181162472,"num_objects":22120,"num_object_clones":0,"num_object_copies":88480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163783,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362123264,"data_stored":362123264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160687,"ondisk_log_size":160687,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753304,"kb_used_data":89928,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93618536,"statfs":{"total":96636764160,"available":95865380864,"internally_reserved":0,"allocated":92086272,"data_stored":91897070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,9,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730620,"kb_used_data":89964,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93641220,"statfs":{"total":96636764160,"available":95888609280,"internally_reserved":0,"allocated":92123136,"data_stored":91933934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594300,"kb_used_data":89236,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93777540,"statfs":{"total":96636764160,"available":96028200960,"internally_reserved":0,"allocated":91377664,"data_stored":91188462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517140178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728768,"kb_used_data":87660,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93643072,"statfs":{"total":96636764160,"available":95890505728,"internally_reserved":0,"allocated":89763840,"data_stored":89574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656469711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89141248,"data_stored":89141248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90771456,"data_stored":90771456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91099136,"data_stored":91099136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91111424,"data_stored":91111424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:02.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:03.424 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-11T05:06:01.882313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173273576,"num_objects":21157,"num_object_clones":0,"num_object_copies":84628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164746,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160950,"ondisk_log_size":160950,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807980,"kb_used_data":341152,"kb_used_omap":14,"kb_used_meta":2466737,"kb_avail":374679380,"statfs":{"total":386547056640,"available":383671685120,"internally_reserved":0,"allocated":349339648,"data_stored":348582840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2525939254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,4,20,16,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18096128,"num_objects":-2209,"num_object_clones":0,"num_object_copies":-8836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2209,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006159"},"pg_stats":[{"pgid":"1.f","version":"35'11159","reported_seq":11254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.170804+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:05:58.170804+0000","last_peered":"2024-09-11T05:05:58.170804+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:05:58.170804+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11100160,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11159,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11378","reported_seq":11492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.937443+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:00.937443+0000","last_peered":"2024-09-11T05:06:00.937443+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:00.937443+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11138208,"num_objects":1360,"num_object_clones":0,"num_object_copies":5440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10864","reported_seq":10967,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.401471+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:05:58.401471+0000","last_peered":"2024-09-11T05:05:58.401471+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:05:58.401471+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10615400,"num_objects":1297,"num_object_clones":0,"num_object_copies":5188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10863,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-914}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11176","reported_seq":11285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.776920+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:00.776920+0000","last_peered":"2024-09-11T05:06:00.776920+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:00.776920+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10155168,"num_objects":1240,"num_object_clones":0,"num_object_copies":4960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11174,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1021}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11200","reported_seq":11324,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.419251+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:05:58.419251+0000","last_peered":"2024-09-11T05:05:58.419251+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:05:58.419251+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10900640,"num_objects":1331,"num_object_clones":0,"num_object_copies":5324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11198,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11337","reported_seq":11443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.474678+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:05:58.474678+0000","last_peered":"2024-09-11T05:05:58.474678+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:05:58.474678+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10674176,"num_objects":1303,"num_object_clones":0,"num_object_copies":5212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11391","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.687956+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:00.687956+0000","last_peered":"2024-09-11T05:06:00.687956+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:00.687956+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10966176,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11389,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-968}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11481","reported_seq":16339,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:58.875827+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:58.875827+0000","last_peered":"2024-09-11T05:05:58.875827+0000","last_clean":"2024-09-11T05:05:58.875827+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:58.875827+0000","last_undegraded":"2024-09-11T05:05:58.875827+0000","last_fullsized":"2024-09-11T05:05:58.875827+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10993664,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11442","reported_seq":11565,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.462883+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:05:58.462883+0000","last_peered":"2024-09-11T05:05:58.462883+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:05:58.462883+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11020608,"num_objects":1346,"num_object_clones":0,"num_object_copies":5384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11243","reported_seq":11356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.287888+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:05:58.287888+0000","last_peered":"2024-09-11T05:05:58.287888+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:05:58.287888+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10523808,"num_objects":1285,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11329","reported_seq":11446,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.525787+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:05:58.525787+0000","last_peered":"2024-09-11T05:05:58.525787+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:05:58.525787+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10841840,"num_objects":1324,"num_object_clones":0,"num_object_copies":5296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-974}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11051","reported_seq":15652,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:58.901989+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:58.901989+0000","last_peered":"2024-09-11T05:05:58.901989+0000","last_clean":"2024-09-11T05:05:58.901989+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:58.901989+0000","last_undegraded":"2024-09-11T05:05:58.901989+0000","last_fullsized":"2024-09-11T05:05:58.901989+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10919936,"num_objects":1333,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4861,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11360","reported_seq":15218,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.480633+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:05:58.480633+0000","last_peered":"2024-09-11T05:05:58.480633+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:05:58.480633+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11164240,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":917}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11254","reported_seq":11361,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.339615+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:05:58.339615+0000","last_peered":"2024-09-11T05:05:58.339615+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:05:58.339615+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10957984,"num_objects":1338,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-926}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11138","reported_seq":11257,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:05:58.487140+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:05:58.487140+0000","last_peered":"2024-09-11T05:05:58.487140+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:05:58.487140+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10409120,"num_objects":1271,"num_object_clones":0,"num_object_copies":5084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11136,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1032}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11447","reported_seq":11567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.842621+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:00.842621+0000","last_peered":"2024-09-11T05:06:00.842621+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:00.842621+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10892448,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11445,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":173273576,"num_objects":21157,"num_object_clones":0,"num_object_copies":84628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164746,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346034176,"data_stored":346034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160950,"ondisk_log_size":160950,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753612,"kb_used_data":86072,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93618228,"statfs":{"total":96636764160,"available":95865065472,"internally_reserved":0,"allocated":88137728,"data_stored":87948526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,9,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730732,"kb_used_data":85912,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93641108,"statfs":{"total":96636764160,"available":95888494592,"internally_reserved":0,"allocated":87973888,"data_stored":87784686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594532,"kb_used_data":85328,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93777308,"statfs":{"total":96636764160,"available":96027963392,"internally_reserved":0,"allocated":87375872,"data_stored":87186670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521400018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,2,3,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729104,"kb_used_data":83840,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93642736,"statfs":{"total":96636764160,"available":95890161664,"internally_reserved":0,"allocated":85852160,"data_stored":85662958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85200896,"data_stored":85200896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86691840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87080960,"data_stored":87080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":87060480,"data_stored":87060480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:03.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:04.057 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:04.074 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-11T05:06:03.883176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":168096232,"num_objects":20525,"num_object_clones":0,"num_object_copies":82100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165378,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808468,"kb_used_data":333284,"kb_used_omap":14,"kb_used_meta":2475057,"kb_avail":374678892,"statfs":{"total":386547056640,"available":383671185408,"internally_reserved":0,"allocated":341282816,"data_stored":340526008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2534458934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,6,6,5,16,16,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21192704,"num_objects":-2587,"num_object_clones":0,"num_object_copies":-10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006146"},"pg_stats":[{"pgid":"1.f","version":"35'11221","reported_seq":11316,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.569272+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:03.569272+0000","last_peered":"2024-09-11T05:06:03.569272+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:03.569272+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10592256,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11378","reported_seq":11492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.937443+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:00.937443+0000","last_peered":"2024-09-11T05:06:00.937443+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:00.937443+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11138208,"num_objects":1360,"num_object_clones":0,"num_object_copies":5440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10925","reported_seq":11030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.488244+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:03.488244+0000","last_peered":"2024-09-11T05:06:03.488244+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:03.488244+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10115688,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10924,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11176","reported_seq":11285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.776920+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:00.776920+0000","last_peered":"2024-09-11T05:06:00.776920+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:00.776920+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10155168,"num_objects":1240,"num_object_clones":0,"num_object_copies":4960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11174,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1021}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11258","reported_seq":11382,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.604368+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:03.604368+0000","last_peered":"2024-09-11T05:06:03.604368+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:03.604368+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10425504,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11256,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11404","reported_seq":11509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.574170+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:03.574170+0000","last_peered":"2024-09-11T05:06:03.574170+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:03.574170+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10125312,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11391","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.687956+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:00.687956+0000","last_peered":"2024-09-11T05:06:00.687956+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:00.687956+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10966176,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11389,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-968}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11481","reported_seq":16339,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:58.875827+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:58.875827+0000","last_peered":"2024-09-11T05:05:58.875827+0000","last_clean":"2024-09-11T05:05:58.875827+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:58.875827+0000","last_undegraded":"2024-09-11T05:05:58.875827+0000","last_fullsized":"2024-09-11T05:05:58.875827+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10993664,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11496","reported_seq":11619,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.497790+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:03.497790+0000","last_peered":"2024-09-11T05:06:03.497790+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:03.497790+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10578240,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11299","reported_seq":11412,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.490950+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:03.490950+0000","last_peered":"2024-09-11T05:06:03.490950+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:03.490950+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065056,"num_objects":1229,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11394","reported_seq":11511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.497155+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:03.497155+0000","last_peered":"2024-09-11T05:06:03.497155+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:03.497155+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10309360,"num_objects":1259,"num_object_clones":0,"num_object_copies":5036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11051","reported_seq":15652,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:58.901989+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:58.901989+0000","last_peered":"2024-09-11T05:05:58.901989+0000","last_clean":"2024-09-11T05:05:58.901989+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:58.901989+0000","last_undegraded":"2024-09-11T05:05:58.901989+0000","last_fullsized":"2024-09-11T05:05:58.901989+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10919936,"num_objects":1333,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4861,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11440","reported_seq":15391,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.558536+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:03.558536+0000","last_peered":"2024-09-11T05:06:03.558536+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:03.558536+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10508880,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11319","reported_seq":11426,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.394932+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:03.394932+0000","last_peered":"2024-09-11T05:06:03.394932+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:03.394932+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10425504,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11317,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11202","reported_seq":11321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.114170+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:03.114170+0000","last_peered":"2024-09-11T05:06:03.114170+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:03.114170+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9884832,"num_objects":1207,"num_object_clones":0,"num_object_copies":4828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11200,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1096}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11447","reported_seq":11567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.842621+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:00.842621+0000","last_peered":"2024-09-11T05:06:00.842621+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:00.842621+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10892448,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11445,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":168096232,"num_objects":20525,"num_object_clones":0,"num_object_copies":82100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165378,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":337952768,"data_stored":337952768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753884,"kb_used_data":82152,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93617956,"statfs":{"total":96636764160,"available":95864786944,"internally_reserved":0,"allocated":84123648,"data_stored":83934446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,3,6,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730948,"kb_used_data":81964,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93640892,"statfs":{"total":96636764160,"available":95888273408,"internally_reserved":0,"allocated":83931136,"data_stored":83741934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,3,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594532,"kb_used_data":85328,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93777308,"statfs":{"total":96636764160,"available":96027963392,"internally_reserved":0,"allocated":87375872,"data_stored":87186670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521400018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,2,3,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729104,"kb_used_data":83840,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93642736,"statfs":{"total":96636764160,"available":95890161664,"internally_reserved":0,"allocated":85852160,"data_stored":85662958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85200896,"data_stored":85200896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86691840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83025920,"data_stored":83025920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83034112,"data_stored":83034112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11221", 'reported_seq': 11316, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.569272+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:03.569272+0000', 'last_peered': '2024-09-11T05:06:03.569272+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:03.569272+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10592256, 'num_objects': 1293, 'num_object_clones': 0, 'num_object_copies': 5172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11221, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11378", 'reported_seq': 11492, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:00.937443+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:00.937443+0000', 'last_peered': '2024-09-11T05:06:00.937443+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:00.937443+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11138208, 'num_objects': 1360, 'num_object_clones': 0, 'num_object_copies': 5440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11376, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10925", 'reported_seq': 11030, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.488244+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:03.488244+0000', 'last_peered': '2024-09-11T05:06:03.488244+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:03.488244+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10115688, 'num_objects': 1236, 'num_object_clones': 0, 'num_object_copies': 4944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1236, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10924, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -975}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11176", 'reported_seq': 11285, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:00.776920+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:06:00.776920+0000', 'last_peered': '2024-09-11T05:06:00.776920+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:00.776920+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10155168, 'num_objects': 1240, 'num_object_clones': 0, 'num_object_copies': 4960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11174, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1021}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:04.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11258", 'reported_seq': 11382, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.604368+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:03.604368+0000', 'last_peered': '2024-09-11T05:06:03.604368+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:03.604368+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10425504, 'num_objects': 1273, 'num_object_clones': 0, 'num_object_copies': 5092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11256, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -948}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11404", 'reported_seq': 11509, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.574170+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:03.574170+0000', 'last_peered': '2024-09-11T05:06:03.574170+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:03.574170+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10125312, 'num_objects': 1236, 'num_object_clones': 0, 'num_object_copies': 4944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11404, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1058}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11391", 'reported_seq': 11486, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:00.687956+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:00.687956+0000', 'last_peered': '2024-09-11T05:06:00.687956+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:00.687956+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10966176, 'num_objects': 1339, 'num_object_clones': 0, 'num_object_copies': 5356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11389, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -968}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:04.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11496", 'reported_seq': 11619, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.497790+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:03.497790+0000', 'last_peered': '2024-09-11T05:06:03.497790+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:03.497790+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10578240, 'num_objects': 1292, 'num_object_clones': 0, 'num_object_copies': 5168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11493, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1074}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11299", 'reported_seq': 11412, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.490950+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:03.490950+0000', 'last_peered': '2024-09-11T05:06:03.490950+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:03.490950+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10065056, 'num_objects': 1229, 'num_object_clones': 0, 'num_object_copies': 4916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11297, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1077}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11394", 'reported_seq': 11511, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.497155+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:03.497155+0000', 'last_peered': '2024-09-11T05:06:03.497155+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:03.497155+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10309360, 'num_objects': 1259, 'num_object_clones': 0, 'num_object_copies': 5036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11391, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1039}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11440", 'reported_seq': 15391, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.558536+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:06:03.558536+0000', 'last_peered': '2024-09-11T05:06:03.558536+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:06:03.558536+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10508880, 'num_objects': 1283, 'num_object_clones': 0, 'num_object_copies': 5132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 400, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1283, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11439, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1934, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11319", 'reported_seq': 11426, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.394932+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:03.394932+0000', 'last_peered': '2024-09-11T05:06:03.394932+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:03.394932+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10425504, 'num_objects': 1273, 'num_object_clones': 0, 'num_object_copies': 5092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11317, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -991}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:04.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11202", 'reported_seq': 11321, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.114170+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:03.114170+0000', 'last_peered': '2024-09-11T05:06:03.114170+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:03.114170+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9884832, 'num_objects': 1207, 'num_object_clones': 0, 'num_object_copies': 4828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11200, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1096}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:04.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11447", 'reported_seq': 11567, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:00.842621+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:00.842621+0000', 'last_peered': '2024-09-11T05:06:00.842621+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:00.842621+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10892448, 'num_objects': 1330, 'num_object_clones': 0, 'num_object_copies': 5320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11445, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -958}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:04.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:04.801 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:04.802 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30091/82100 objects degraded (36.652%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":691,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20525,"data_bytes":168096232,"bytes_used":2875871232,"bytes_avail":383671185408,"bytes_total":386547056640,"degraded_objects":30091,"degraded_total":82100,"degraded_ratio":0.36651644336175404,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":215},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:04.831 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:06:04.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:05.451 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:05.489 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-11T05:06:03.883176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":168096232,"num_objects":20525,"num_object_clones":0,"num_object_copies":82100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165378,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808468,"kb_used_data":333284,"kb_used_omap":14,"kb_used_meta":2475057,"kb_avail":374678892,"statfs":{"total":386547056640,"available":383671185408,"internally_reserved":0,"allocated":341282816,"data_stored":340526008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2534458934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,6,6,5,16,16,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21192704,"num_objects":-2587,"num_object_clones":0,"num_object_copies":-10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006146"},"pg_stats":[{"pgid":"1.f","version":"35'11221","reported_seq":11316,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.569272+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:03.569272+0000","last_peered":"2024-09-11T05:06:03.569272+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:03.569272+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10592256,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11378","reported_seq":11492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.937443+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:00.937443+0000","last_peered":"2024-09-11T05:06:00.937443+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:00.937443+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11138208,"num_objects":1360,"num_object_clones":0,"num_object_copies":5440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10925","reported_seq":11030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.488244+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:03.488244+0000","last_peered":"2024-09-11T05:06:03.488244+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:03.488244+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10115688,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10924,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11176","reported_seq":11285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.776920+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:00.776920+0000","last_peered":"2024-09-11T05:06:00.776920+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:00.776920+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10155168,"num_objects":1240,"num_object_clones":0,"num_object_copies":4960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11174,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1021}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11258","reported_seq":11382,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.604368+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:03.604368+0000","last_peered":"2024-09-11T05:06:03.604368+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:03.604368+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10425504,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11256,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11404","reported_seq":11509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.574170+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:03.574170+0000","last_peered":"2024-09-11T05:06:03.574170+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:03.574170+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10125312,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11391","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.687956+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:00.687956+0000","last_peered":"2024-09-11T05:06:00.687956+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:00.687956+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10966176,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11389,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-968}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11481","reported_seq":16339,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:58.875827+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:05:58.875827+0000","last_peered":"2024-09-11T05:05:58.875827+0000","last_clean":"2024-09-11T05:05:58.875827+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:05:58.875827+0000","last_undegraded":"2024-09-11T05:05:58.875827+0000","last_fullsized":"2024-09-11T05:05:58.875827+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10993664,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11496","reported_seq":11619,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.497790+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:03.497790+0000","last_peered":"2024-09-11T05:06:03.497790+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:03.497790+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10578240,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11299","reported_seq":11412,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.490950+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:03.490950+0000","last_peered":"2024-09-11T05:06:03.490950+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:03.490950+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065056,"num_objects":1229,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11394","reported_seq":11511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.497155+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:03.497155+0000","last_peered":"2024-09-11T05:06:03.497155+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:03.497155+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10309360,"num_objects":1259,"num_object_clones":0,"num_object_copies":5036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11051","reported_seq":15652,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:05:58.901989+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:05:58.901989+0000","last_peered":"2024-09-11T05:05:58.901989+0000","last_clean":"2024-09-11T05:05:58.901989+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:05:58.901989+0000","last_undegraded":"2024-09-11T05:05:58.901989+0000","last_fullsized":"2024-09-11T05:05:58.901989+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10919936,"num_objects":1333,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4861,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11440","reported_seq":15391,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.558536+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:03.558536+0000","last_peered":"2024-09-11T05:06:03.558536+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:03.558536+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10508880,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11319","reported_seq":11426,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.394932+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:03.394932+0000","last_peered":"2024-09-11T05:06:03.394932+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:03.394932+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10425504,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11317,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11202","reported_seq":11321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.114170+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:03.114170+0000","last_peered":"2024-09-11T05:06:03.114170+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:03.114170+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9884832,"num_objects":1207,"num_object_clones":0,"num_object_copies":4828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11200,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1096}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11447","reported_seq":11567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:00.842621+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:00.842621+0000","last_peered":"2024-09-11T05:06:00.842621+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:00.842621+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10892448,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11445,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":168096232,"num_objects":20525,"num_object_clones":0,"num_object_copies":82100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165378,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":337952768,"data_stored":337952768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753884,"kb_used_data":82152,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93617956,"statfs":{"total":96636764160,"available":95864786944,"internally_reserved":0,"allocated":84123648,"data_stored":83934446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,3,6,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730948,"kb_used_data":81964,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93640892,"statfs":{"total":96636764160,"available":95888273408,"internally_reserved":0,"allocated":83931136,"data_stored":83741934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,3,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594532,"kb_used_data":85328,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93777308,"statfs":{"total":96636764160,"available":96027963392,"internally_reserved":0,"allocated":87375872,"data_stored":87186670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":521400018},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,2,3,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729104,"kb_used_data":83840,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93642736,"statfs":{"total":96636764160,"available":95890161664,"internally_reserved":0,"allocated":85852160,"data_stored":85662958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85200896,"data_stored":85200896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86691840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83025920,"data_stored":83025920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83034112,"data_stored":83034112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:08.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:09.166 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:09.167 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-11T05:06:07.885608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165106152,"num_objects":20160,"num_object_clones":0,"num_object_copies":80640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165743,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792420,"kb_used_data":325548,"kb_used_omap":14,"kb_used_meta":2466737,"kb_avail":374694940,"statfs":{"total":386547056640,"available":383687618560,"internally_reserved":0,"allocated":333361152,"data_stored":332604344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2525939254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,6,4,6,16,14,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18219008,"num_objects":-2224,"num_object_clones":0,"num_object_copies":-8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007436"},"pg_stats":[{"pgid":"1.f","version":"35'11221","reported_seq":11316,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.569272+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:03.569272+0000","last_peered":"2024-09-11T05:06:03.569272+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:03.569272+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10592256,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11442","reported_seq":11557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.919786+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:05.919786+0000","last_peered":"2024-09-11T05:06:05.919786+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:05.919786+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613920,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11440,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10925","reported_seq":11030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.488244+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:03.488244+0000","last_peered":"2024-09-11T05:06:03.488244+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:03.488244+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10115688,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10924,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11243","reported_seq":11351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.820055+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:05.820055+0000","last_peered":"2024-09-11T05:06:05.820055+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:05.820055+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9606304,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11258","reported_seq":11382,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.604368+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:03.604368+0000","last_peered":"2024-09-11T05:06:03.604368+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:03.604368+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10425504,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11256,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11404","reported_seq":11509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.574170+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:03.574170+0000","last_peered":"2024-09-11T05:06:03.574170+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:03.574170+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10125312,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11454","reported_seq":11548,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.881460+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:05.881460+0000","last_peered":"2024-09-11T05:06:05.881460+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:05.881460+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10450080,"num_objects":1276,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1031}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11534","reported_seq":16392,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:04.283227+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:04.283227+0000","last_peered":"2024-09-11T05:06:04.283227+0000","last_clean":"2024-09-11T05:06:04.283227+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:04.283227+0000","last_undegraded":"2024-09-11T05:06:04.283227+0000","last_fullsized":"2024-09-11T05:06:04.283227+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10559488,"num_objects":1289,"num_object_clones":0,"num_object_copies":5156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2244,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11496","reported_seq":11619,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.497790+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:03.497790+0000","last_peered":"2024-09-11T05:06:03.497790+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:03.497790+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10578240,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11299","reported_seq":11412,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.490950+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:03.490950+0000","last_peered":"2024-09-11T05:06:03.490950+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:03.490950+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065056,"num_objects":1229,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11394","reported_seq":11511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.497155+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:03.497155+0000","last_peered":"2024-09-11T05:06:03.497155+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:03.497155+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10309360,"num_objects":1259,"num_object_clones":0,"num_object_copies":5036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11092","reported_seq":15693,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:03.774151+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:03.774151+0000","last_peered":"2024-09-11T05:06:03.774151+0000","last_clean":"2024-09-11T05:06:03.774151+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:03.774151+0000","last_undegraded":"2024-09-11T05:06:03.774151+0000","last_fullsized":"2024-09-11T05:06:03.774151+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10584064,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4902,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11440","reported_seq":15391,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.558536+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:03.558536+0000","last_peered":"2024-09-11T05:06:03.558536+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:03.558536+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10508880,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11319","reported_seq":11426,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.394932+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:03.394932+0000","last_peered":"2024-09-11T05:06:03.394932+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:03.394932+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10425504,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11317,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11202","reported_seq":11321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.114170+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:03.114170+0000","last_peered":"2024-09-11T05:06:03.114170+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:03.114170+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9884832,"num_objects":1207,"num_object_clones":0,"num_object_copies":4828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11200,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1096}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11524","reported_seq":11644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.907323+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:05.907323+0000","last_peered":"2024-09-11T05:06:05.907323+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:05.907323+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10261664,"num_objects":1253,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165106152,"num_objects":20160,"num_object_clones":0,"num_object_copies":80640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165743,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329940992,"data_stored":329940992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753884,"kb_used_data":82152,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93617956,"statfs":{"total":96636764160,"available":95864786944,"internally_reserved":0,"allocated":84123648,"data_stored":83934446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,3,6,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730948,"kb_used_data":81964,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93640892,"statfs":{"total":96636764160,"available":95888273408,"internally_reserved":0,"allocated":83931136,"data_stored":83741934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,3,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578176,"kb_used_data":81456,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93793664,"statfs":{"total":96636764160,"available":96044711936,"internally_reserved":0,"allocated":83410944,"data_stored":83221742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":508620498},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729412,"kb_used_data":79976,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93642428,"statfs":{"total":96636764160,"available":95889846272,"internally_reserved":0,"allocated":81895424,"data_stored":81706222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81244160,"data_stored":81244160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82636800,"data_stored":82636800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83025920,"data_stored":83025920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83034112,"data_stored":83034112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:09.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:09.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:09.836 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:09.864 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-11T05:06:07.885608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165106152,"num_objects":20160,"num_object_clones":0,"num_object_copies":80640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165743,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792420,"kb_used_data":325548,"kb_used_omap":14,"kb_used_meta":2466737,"kb_avail":374694940,"statfs":{"total":386547056640,"available":383687618560,"internally_reserved":0,"allocated":333361152,"data_stored":332604344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2525939254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,6,4,6,16,14,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18219008,"num_objects":-2224,"num_object_clones":0,"num_object_copies":-8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007436"},"pg_stats":[{"pgid":"1.f","version":"35'11221","reported_seq":11316,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.569272+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:03.569272+0000","last_peered":"2024-09-11T05:06:03.569272+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:03.569272+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10592256,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11442","reported_seq":11557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.919786+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:05.919786+0000","last_peered":"2024-09-11T05:06:05.919786+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:05.919786+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613920,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11440,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10925","reported_seq":11030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.488244+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:03.488244+0000","last_peered":"2024-09-11T05:06:03.488244+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:03.488244+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10115688,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10924,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11243","reported_seq":11351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.820055+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:05.820055+0000","last_peered":"2024-09-11T05:06:05.820055+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:05.820055+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9606304,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11258","reported_seq":11382,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.604368+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:03.604368+0000","last_peered":"2024-09-11T05:06:03.604368+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:03.604368+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10425504,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11256,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11404","reported_seq":11509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.574170+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:03.574170+0000","last_peered":"2024-09-11T05:06:03.574170+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:03.574170+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10125312,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11454","reported_seq":11548,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.881460+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:05.881460+0000","last_peered":"2024-09-11T05:06:05.881460+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:05.881460+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10450080,"num_objects":1276,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1031}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11534","reported_seq":16392,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:04.283227+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:04.283227+0000","last_peered":"2024-09-11T05:06:04.283227+0000","last_clean":"2024-09-11T05:06:04.283227+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:04.283227+0000","last_undegraded":"2024-09-11T05:06:04.283227+0000","last_fullsized":"2024-09-11T05:06:04.283227+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10559488,"num_objects":1289,"num_object_clones":0,"num_object_copies":5156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2244,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11496","reported_seq":11619,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.497790+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:03.497790+0000","last_peered":"2024-09-11T05:06:03.497790+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:03.497790+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10578240,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11299","reported_seq":11412,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.490950+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:03.490950+0000","last_peered":"2024-09-11T05:06:03.490950+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:03.490950+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065056,"num_objects":1229,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11394","reported_seq":11511,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.497155+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:03.497155+0000","last_peered":"2024-09-11T05:06:03.497155+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:03.497155+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10309360,"num_objects":1259,"num_object_clones":0,"num_object_copies":5036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11092","reported_seq":15693,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:03.774151+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:03.774151+0000","last_peered":"2024-09-11T05:06:03.774151+0000","last_clean":"2024-09-11T05:06:03.774151+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:03.774151+0000","last_undegraded":"2024-09-11T05:06:03.774151+0000","last_fullsized":"2024-09-11T05:06:03.774151+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10584064,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4902,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11440","reported_seq":15391,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.558536+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:03.558536+0000","last_peered":"2024-09-11T05:06:03.558536+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:03.558536+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10508880,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11319","reported_seq":11426,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.394932+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:03.394932+0000","last_peered":"2024-09-11T05:06:03.394932+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:03.394932+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10425504,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11317,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-991}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11202","reported_seq":11321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:03.114170+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:03.114170+0000","last_peered":"2024-09-11T05:06:03.114170+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:03.114170+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9884832,"num_objects":1207,"num_object_clones":0,"num_object_copies":4828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11200,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1096}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11524","reported_seq":11644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.907323+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:05.907323+0000","last_peered":"2024-09-11T05:06:05.907323+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:05.907323+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10261664,"num_objects":1253,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165106152,"num_objects":20160,"num_object_clones":0,"num_object_copies":80640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165743,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329940992,"data_stored":329940992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753884,"kb_used_data":82152,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93617956,"statfs":{"total":96636764160,"available":95864786944,"internally_reserved":0,"allocated":84123648,"data_stored":83934446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,3,6,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730948,"kb_used_data":81964,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93640892,"statfs":{"total":96636764160,"available":95888273408,"internally_reserved":0,"allocated":83931136,"data_stored":83741934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,3,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578176,"kb_used_data":81456,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93793664,"statfs":{"total":96636764160,"available":96044711936,"internally_reserved":0,"allocated":83410944,"data_stored":83221742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":508620498},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729412,"kb_used_data":79976,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93642428,"statfs":{"total":96636764160,"available":95889846272,"internally_reserved":0,"allocated":81895424,"data_stored":81706222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81244160,"data_stored":81244160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82636800,"data_stored":82636800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83025920,"data_stored":83025920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83034112,"data_stored":83034112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11221", 'reported_seq': 11316, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.569272+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:03.569272+0000', 'last_peered': '2024-09-11T05:06:03.569272+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:03.569272+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10592256, 'num_objects': 1293, 'num_object_clones': 0, 'num_object_copies': 5172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11221, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11442", 'reported_seq': 11557, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:05.919786+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:05.919786+0000', 'last_peered': '2024-09-11T05:06:05.919786+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:05.919786+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10613920, 'num_objects': 1296, 'num_object_clones': 0, 'num_object_copies': 5184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11440, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10925", 'reported_seq': 11030, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.488244+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:03.488244+0000', 'last_peered': '2024-09-11T05:06:03.488244+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:03.488244+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10115688, 'num_objects': 1236, 'num_object_clones': 0, 'num_object_copies': 4944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1236, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10924, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -975}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11243", 'reported_seq': 11351, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:05.820055+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:06:05.820055+0000', 'last_peered': '2024-09-11T05:06:05.820055+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:05.820055+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9606304, 'num_objects': 1173, 'num_object_clones': 0, 'num_object_copies': 4692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11241, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:09.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11258", 'reported_seq': 11382, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.604368+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:03.604368+0000', 'last_peered': '2024-09-11T05:06:03.604368+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:03.604368+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10425504, 'num_objects': 1273, 'num_object_clones': 0, 'num_object_copies': 5092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11256, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -948}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11404", 'reported_seq': 11509, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.574170+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:03.574170+0000', 'last_peered': '2024-09-11T05:06:03.574170+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:03.574170+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10125312, 'num_objects': 1236, 'num_object_clones': 0, 'num_object_copies': 4944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11404, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1058}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11454", 'reported_seq': 11548, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:05.881460+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:05.881460+0000', 'last_peered': '2024-09-11T05:06:05.881460+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:05.881460+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10450080, 'num_objects': 1276, 'num_object_clones': 0, 'num_object_copies': 5104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11452, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1031}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11496", 'reported_seq': 11619, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.497790+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:03.497790+0000', 'last_peered': '2024-09-11T05:06:03.497790+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:03.497790+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10578240, 'num_objects': 1292, 'num_object_clones': 0, 'num_object_copies': 5168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11493, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1074}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11299", 'reported_seq': 11412, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.490950+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:03.490950+0000', 'last_peered': '2024-09-11T05:06:03.490950+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:03.490950+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10065056, 'num_objects': 1229, 'num_object_clones': 0, 'num_object_copies': 4916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11297, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1077}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11394", 'reported_seq': 11511, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.497155+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:03.497155+0000', 'last_peered': '2024-09-11T05:06:03.497155+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:03.497155+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10309360, 'num_objects': 1259, 'num_object_clones': 0, 'num_object_copies': 5036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11391, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1039}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:09.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:06:09.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11440", 'reported_seq': 15391, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.558536+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:06:03.558536+0000', 'last_peered': '2024-09-11T05:06:03.558536+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:06:03.558536+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10508880, 'num_objects': 1283, 'num_object_clones': 0, 'num_object_copies': 5132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 400, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1283, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11439, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1934, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:09.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:09.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11319", 'reported_seq': 11426, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.394932+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:03.394932+0000', 'last_peered': '2024-09-11T05:06:03.394932+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:03.394932+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10425504, 'num_objects': 1273, 'num_object_clones': 0, 'num_object_copies': 5092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11317, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -991}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:09.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:09.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11202", 'reported_seq': 11321, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:03.114170+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:03.114170+0000', 'last_peered': '2024-09-11T05:06:03.114170+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:03.114170+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9884832, 'num_objects': 1207, 'num_object_clones': 0, 'num_object_copies': 4828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11200, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1096}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:09.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:09.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11524", 'reported_seq': 11644, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:05.907323+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:05.907323+0000', 'last_peered': '2024-09-11T05:06:05.907323+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:05.907323+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10261664, 'num_objects': 1253, 'num_object_clones': 0, 'num_object_copies': 5012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11522, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1035}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:09.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:10.620 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:10.620 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30033/77804 objects degraded (38.601%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":697,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":19451,"data_bytes":159298024,"bytes_used":2842664960,"bytes_avail":383704391680,"bytes_total":386547056640,"degraded_objects":30033,"degraded_total":77804,"degraded_ratio":0.38600843144311348,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":222},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:10.645 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:06:10.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:11.283 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:11.284 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:11.310 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-11T05:06:09.886451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":159298024,"num_objects":19451,"num_object_clones":0,"num_object_copies":77804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166452,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2776040,"kb_used_data":313372,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374711320,"statfs":{"total":386547056640,"available":383704391680,"internally_reserved":0,"allocated":320892928,"data_stored":320136120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2521679414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,6,7,4,14,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21864448,"num_objects":-2669,"num_object_clones":0,"num_object_copies":-10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2669,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007613"},"pg_stats":[{"pgid":"1.f","version":"35'11281","reported_seq":11376,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.551991+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:08.551991+0000","last_peered":"2024-09-11T05:06:08.551991+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:08.551991+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10100736,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11442","reported_seq":11557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.919786+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:05.919786+0000","last_peered":"2024-09-11T05:06:05.919786+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:05.919786+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613920,"num_objects":1296,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11440,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10979","reported_seq":11084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.613094+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:08.613094+0000","last_peered":"2024-09-11T05:06:08.613094+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:08.613094+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9673320,"num_objects":1182,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10978,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1029}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11243","reported_seq":11351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.820055+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:05.820055+0000","last_peered":"2024-09-11T05:06:05.820055+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:05.820055+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9606304,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11327","reported_seq":11451,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.573323+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:08.573323+0000","last_peered":"2024-09-11T05:06:08.573323+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:08.573323+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9860256,"num_objects":1204,"num_object_clones":0,"num_object_copies":4816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11451","reported_seq":11557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.549643+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:08.549643+0000","last_peered":"2024-09-11T05:06:08.549643+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:08.549643+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9740288,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11451,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11454","reported_seq":11548,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.881460+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:05.881460+0000","last_peered":"2024-09-11T05:06:05.881460+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:05.881460+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10450080,"num_objects":1276,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1031}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11600","reported_seq":16458,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:09.243486+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:09.243486+0000","last_peered":"2024-09-11T05:06:09.243486+0000","last_clean":"2024-09-11T05:06:09.243486+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:09.243486+0000","last_undegraded":"2024-09-11T05:06:09.243486+0000","last_fullsized":"2024-09-11T05:06:09.243486+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10018816,"num_objects":1223,"num_object_clones":0,"num_object_copies":4892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11555","reported_seq":11679,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.528447+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:08.528447+0000","last_peered":"2024-09-11T05:06:08.528447+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:08.528447+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10094912,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11552,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11374","reported_seq":11487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.610787+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:08.610787+0000","last_peered":"2024-09-11T05:06:08.610787+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:08.610787+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9450656,"num_objects":1154,"num_object_clones":0,"num_object_copies":4616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11372,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1152}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11453","reported_seq":11571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.516814+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:08.516814+0000","last_peered":"2024-09-11T05:06:08.516814+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:08.516814+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9826032,"num_objects":1200,"num_object_clones":0,"num_object_copies":4800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11145","reported_seq":15747,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:09.362060+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:09.362060+0000","last_peered":"2024-09-11T05:06:09.362060+0000","last_clean":"2024-09-11T05:06:09.362060+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:09.362060+0000","last_undegraded":"2024-09-11T05:06:09.362060+0000","last_fullsized":"2024-09-11T05:06:09.362060+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10149888,"num_objects":1239,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4955,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11505","reported_seq":15573,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.620074+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:08.620074+0000","last_peered":"2024-09-11T05:06:08.620074+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:08.620074+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9976400,"num_objects":1218,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11504,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11363","reported_seq":11470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.655970+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:08.655970+0000","last_peered":"2024-09-11T05:06:08.655970+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:08.655970+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065056,"num_objects":1229,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11361,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11260","reported_seq":11379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:08.374733+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:08.374733+0000","last_peered":"2024-09-11T05:06:08.374733+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:08.374733+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9409696,"num_objects":1149,"num_object_clones":0,"num_object_copies":4596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11258,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11524","reported_seq":11644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:05.907323+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:05.907323+0000","last_peered":"2024-09-11T05:06:05.907323+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:05.907323+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10261664,"num_objects":1253,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":159298024,"num_objects":19451,"num_object_clones":0,"num_object_copies":77804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166452,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":318058496,"data_stored":318058496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736824,"kb_used_data":77608,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93635016,"statfs":{"total":96636764160,"available":95882256384,"internally_reserved":0,"allocated":79470592,"data_stored":79281390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,1,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731344,"kb_used_data":78208,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93640496,"statfs":{"total":96636764160,"available":95887867904,"internally_reserved":0,"allocated":80084992,"data_stored":79895790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1,1,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578460,"kb_used_data":77580,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93793380,"statfs":{"total":96636764160,"available":96044421120,"internally_reserved":0,"allocated":79441920,"data_stored":79252718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512880338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729412,"kb_used_data":79976,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93642428,"statfs":{"total":96636764160,"available":95889846272,"internally_reserved":0,"allocated":81895424,"data_stored":81706222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81244160,"data_stored":81244160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78692352,"data_stored":78692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79060992,"data_stored":79060992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":79060992,"data_stored":79060992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:14.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:14.969 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:14.970 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:14.998 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-11T05:06:13.888245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":152596968,"num_objects":18633,"num_object_clones":0,"num_object_copies":74532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167270,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2760480,"kb_used_data":301976,"kb_used_omap":14,"kb_used_meta":2458417,"kb_avail":374726880,"statfs":{"total":386547056640,"available":383720325120,"internally_reserved":0,"allocated":309223424,"data_stored":308466616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2517419574},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,8,6,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20676608,"num_objects":-2524,"num_object_clones":0,"num_object_copies":-10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005931"},"pg_stats":[{"pgid":"1.f","version":"35'11337","reported_seq":11432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.591246+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:13.591246+0000","last_peered":"2024-09-11T05:06:13.591246+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:13.591246+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9641984,"num_objects":1177,"num_object_clones":0,"num_object_copies":4708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11511","reported_seq":11625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.938623+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:10.938623+0000","last_peered":"2024-09-11T05:06:10.938623+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:10.938623+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10048672,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1031}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11045","reported_seq":11150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.660287+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:13.660287+0000","last_peered":"2024-09-11T05:06:13.660287+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:13.660287+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9132648,"num_objects":1116,"num_object_clones":0,"num_object_copies":4464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11044,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11296","reported_seq":11405,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.949497+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:10.949497+0000","last_peered":"2024-09-11T05:06:10.949497+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:10.949497+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9172128,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11383","reported_seq":11507,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.767907+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:13.767907+0000","last_peered":"2024-09-11T05:06:13.767907+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:13.767907+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9401504,"num_objects":1148,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1073}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11506","reported_seq":11612,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.719315+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:13.719315+0000","last_peered":"2024-09-11T05:06:13.719315+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:13.719315+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9289728,"num_objects":1134,"num_object_clones":0,"num_object_copies":4536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1160}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11515","reported_seq":11609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.890117+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:10.890117+0000","last_peered":"2024-09-11T05:06:10.890117+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:10.890117+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9950368,"num_objects":1215,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11513,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11600","reported_seq":16458,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:09.243486+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:09.243486+0000","last_peered":"2024-09-11T05:06:09.243486+0000","last_clean":"2024-09-11T05:06:09.243486+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:09.243486+0000","last_undegraded":"2024-09-11T05:06:09.243486+0000","last_fullsized":"2024-09-11T05:06:09.243486+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10018816,"num_objects":1223,"num_object_clones":0,"num_object_copies":4892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11609","reported_seq":11732,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.712163+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:13.712163+0000","last_peered":"2024-09-11T05:06:13.712163+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:13.712163+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9652544,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11606,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11436","reported_seq":11549,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.655813+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:13.655813+0000","last_peered":"2024-09-11T05:06:13.655813+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:13.655813+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942752,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11434,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11497","reported_seq":11615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.499829+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:13.499829+0000","last_peered":"2024-09-11T05:06:13.499829+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:13.499829+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9465584,"num_objects":1156,"num_object_clones":0,"num_object_copies":4624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11494,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11145","reported_seq":15747,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:09.362060+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:09.362060+0000","last_peered":"2024-09-11T05:06:09.362060+0000","last_clean":"2024-09-11T05:06:09.362060+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:09.362060+0000","last_undegraded":"2024-09-11T05:06:09.362060+0000","last_fullsized":"2024-09-11T05:06:09.362060+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10149888,"num_objects":1239,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4955,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11559","reported_seq":15749,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.700355+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:13.700355+0000","last_peered":"2024-09-11T05:06:13.700355+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:13.700355+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9534032,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2053,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11412","reported_seq":11519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.551462+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:13.551462+0000","last_peered":"2024-09-11T05:06:13.551462+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:13.551462+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9663648,"num_objects":1180,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11410,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1084}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11327","reported_seq":11443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.493289+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:13.493289+0000","last_peered":"2024-09-11T05:06:13.493289+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:13.493289+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8860832,"num_objects":1082,"num_object_clones":0,"num_object_copies":4328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11596","reported_seq":11716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.969424+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:10.969424+0000","last_peered":"2024-09-11T05:06:10.969424+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:10.969424+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9671840,"num_objects":1181,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":152596968,"num_objects":18633,"num_object_clones":0,"num_object_copies":74532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167270,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":306298880,"data_stored":306298880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737168,"kb_used_data":73792,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93634672,"statfs":{"total":96636764160,"available":95881904128,"internally_reserved":0,"allocated":75563008,"data_stored":75373806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715048,"kb_used_data":74384,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93656792,"statfs":{"total":96636764160,"available":95904555008,"internally_reserved":0,"allocated":76169216,"data_stored":75980014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578460,"kb_used_data":77580,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93793380,"statfs":{"total":96636764160,"available":96044421120,"internally_reserved":0,"allocated":79441920,"data_stored":79252718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512880338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729804,"kb_used_data":76220,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93642036,"statfs":{"total":96636764160,"available":95889444864,"internally_reserved":0,"allocated":78049280,"data_stored":77860078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77373440,"data_stored":77373440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78692352,"data_stored":78692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75116544,"data_stored":75116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75116544,"data_stored":75116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:14.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:15.600 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:15.600 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:15.625 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-11T05:06:13.888245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":152596968,"num_objects":18633,"num_object_clones":0,"num_object_copies":74532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167270,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2760480,"kb_used_data":301976,"kb_used_omap":14,"kb_used_meta":2458417,"kb_avail":374726880,"statfs":{"total":386547056640,"available":383720325120,"internally_reserved":0,"allocated":309223424,"data_stored":308466616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2517419574},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,8,6,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20676608,"num_objects":-2524,"num_object_clones":0,"num_object_copies":-10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005931"},"pg_stats":[{"pgid":"1.f","version":"35'11337","reported_seq":11432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.591246+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:13.591246+0000","last_peered":"2024-09-11T05:06:13.591246+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:13.591246+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9641984,"num_objects":1177,"num_object_clones":0,"num_object_copies":4708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11511","reported_seq":11625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.938623+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:10.938623+0000","last_peered":"2024-09-11T05:06:10.938623+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:10.938623+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10048672,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1031}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11045","reported_seq":11150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.660287+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:13.660287+0000","last_peered":"2024-09-11T05:06:13.660287+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:13.660287+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9132648,"num_objects":1116,"num_object_clones":0,"num_object_copies":4464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11044,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11296","reported_seq":11405,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.949497+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:10.949497+0000","last_peered":"2024-09-11T05:06:10.949497+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:10.949497+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9172128,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11383","reported_seq":11507,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.767907+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:13.767907+0000","last_peered":"2024-09-11T05:06:13.767907+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:13.767907+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9401504,"num_objects":1148,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1073}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11506","reported_seq":11612,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.719315+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:13.719315+0000","last_peered":"2024-09-11T05:06:13.719315+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:13.719315+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9289728,"num_objects":1134,"num_object_clones":0,"num_object_copies":4536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1160}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11515","reported_seq":11609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.890117+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:10.890117+0000","last_peered":"2024-09-11T05:06:10.890117+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:10.890117+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9950368,"num_objects":1215,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11513,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11600","reported_seq":16458,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:09.243486+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:09.243486+0000","last_peered":"2024-09-11T05:06:09.243486+0000","last_clean":"2024-09-11T05:06:09.243486+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:09.243486+0000","last_undegraded":"2024-09-11T05:06:09.243486+0000","last_fullsized":"2024-09-11T05:06:09.243486+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10018816,"num_objects":1223,"num_object_clones":0,"num_object_copies":4892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11609","reported_seq":11732,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.712163+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:13.712163+0000","last_peered":"2024-09-11T05:06:13.712163+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:13.712163+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9652544,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11606,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11436","reported_seq":11549,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.655813+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:13.655813+0000","last_peered":"2024-09-11T05:06:13.655813+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:13.655813+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942752,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11434,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11497","reported_seq":11615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.499829+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:13.499829+0000","last_peered":"2024-09-11T05:06:13.499829+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:13.499829+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9465584,"num_objects":1156,"num_object_clones":0,"num_object_copies":4624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11494,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11145","reported_seq":15747,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:09.362060+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:09.362060+0000","last_peered":"2024-09-11T05:06:09.362060+0000","last_clean":"2024-09-11T05:06:09.362060+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:09.362060+0000","last_undegraded":"2024-09-11T05:06:09.362060+0000","last_fullsized":"2024-09-11T05:06:09.362060+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10149888,"num_objects":1239,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4955,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11559","reported_seq":15749,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.700355+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:13.700355+0000","last_peered":"2024-09-11T05:06:13.700355+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:13.700355+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9534032,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2053,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11412","reported_seq":11519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.551462+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:13.551462+0000","last_peered":"2024-09-11T05:06:13.551462+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:13.551462+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9663648,"num_objects":1180,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11410,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1084}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11327","reported_seq":11443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.493289+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:13.493289+0000","last_peered":"2024-09-11T05:06:13.493289+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:13.493289+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8860832,"num_objects":1082,"num_object_clones":0,"num_object_copies":4328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11596","reported_seq":11716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.969424+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:10.969424+0000","last_peered":"2024-09-11T05:06:10.969424+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:10.969424+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9671840,"num_objects":1181,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":152596968,"num_objects":18633,"num_object_clones":0,"num_object_copies":74532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167270,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":306298880,"data_stored":306298880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737168,"kb_used_data":73792,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93634672,"statfs":{"total":96636764160,"available":95881904128,"internally_reserved":0,"allocated":75563008,"data_stored":75373806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715048,"kb_used_data":74384,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93656792,"statfs":{"total":96636764160,"available":95904555008,"internally_reserved":0,"allocated":76169216,"data_stored":75980014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578460,"kb_used_data":77580,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93793380,"statfs":{"total":96636764160,"available":96044421120,"internally_reserved":0,"allocated":79441920,"data_stored":79252718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":512880338},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729804,"kb_used_data":76220,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93642036,"statfs":{"total":96636764160,"available":95889444864,"internally_reserved":0,"allocated":78049280,"data_stored":77860078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77373440,"data_stored":77373440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78692352,"data_stored":78692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75116544,"data_stored":75116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75116544,"data_stored":75116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:15.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:15.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11337", 'reported_seq': 11432, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.591246+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:13.591246+0000', 'last_peered': '2024-09-11T05:06:13.591246+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:13.591246+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9641984, 'num_objects': 1177, 'num_object_clones': 0, 'num_object_copies': 4708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11337, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1114}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:15.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:15.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11511", 'reported_seq': 11625, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:10.938623+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:10.938623+0000', 'last_peered': '2024-09-11T05:06:10.938623+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:10.938623+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10048672, 'num_objects': 1227, 'num_object_clones': 0, 'num_object_copies': 4908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11509, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1031}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:15.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:15.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11045", 'reported_seq': 11150, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.660287+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:13.660287+0000', 'last_peered': '2024-09-11T05:06:13.660287+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:13.660287+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9132648, 'num_objects': 1116, 'num_object_clones': 0, 'num_object_copies': 4464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1116, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11044, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1095}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:15.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:15.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11296", 'reported_seq': 11405, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:10.949497+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:06:10.949497+0000', 'last_peered': '2024-09-11T05:06:10.949497+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:10.949497+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9172128, 'num_objects': 1120, 'num_object_clones': 0, 'num_object_copies': 4480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11294, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:15.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:15.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11383", 'reported_seq': 11507, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.767907+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:13.767907+0000', 'last_peered': '2024-09-11T05:06:13.767907+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:13.767907+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9401504, 'num_objects': 1148, 'num_object_clones': 0, 'num_object_copies': 4592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11381, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1073}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:15.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:15.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11506", 'reported_seq': 11612, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.719315+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:13.719315+0000', 'last_peered': '2024-09-11T05:06:13.719315+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:13.719315+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9289728, 'num_objects': 1134, 'num_object_clones': 0, 'num_object_copies': 4536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11506, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1160}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:15.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:15.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11515", 'reported_seq': 11609, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:10.890117+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:10.890117+0000', 'last_peered': '2024-09-11T05:06:10.890117+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:10.890117+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9950368, 'num_objects': 1215, 'num_object_clones': 0, 'num_object_copies': 4860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11513, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1092}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:15.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:15.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11609", 'reported_seq': 11732, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.712163+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:13.712163+0000', 'last_peered': '2024-09-11T05:06:13.712163+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:13.712163+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9652544, 'num_objects': 1179, 'num_object_clones': 0, 'num_object_copies': 4716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11606, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1187}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:15.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:15.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11436", 'reported_seq': 11549, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.655813+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:13.655813+0000', 'last_peered': '2024-09-11T05:06:13.655813+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:13.655813+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8942752, 'num_objects': 1092, 'num_object_clones': 0, 'num_object_copies': 4368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11434, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1214}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:15.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11497", 'reported_seq': 11615, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.499829+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:13.499829+0000', 'last_peered': '2024-09-11T05:06:13.499829+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:13.499829+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9465584, 'num_objects': 1156, 'num_object_clones': 0, 'num_object_copies': 4624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11494, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1142}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11559", 'reported_seq': 15749, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.700355+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:06:13.700355+0000', 'last_peered': '2024-09-11T05:06:13.700355+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:06:13.700355+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9534032, 'num_objects': 1164, 'num_object_clones': 0, 'num_object_copies': 4656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11558, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2053, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11412", 'reported_seq': 11519, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.551462+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:13.551462+0000', 'last_peered': '2024-09-11T05:06:13.551462+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:13.551462+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9663648, 'num_objects': 1180, 'num_object_clones': 0, 'num_object_copies': 4720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11410, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1084}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11327", 'reported_seq': 11443, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:13.493289+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:13.493289+0000', 'last_peered': '2024-09-11T05:06:13.493289+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:13.493289+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8860832, 'num_objects': 1082, 'num_object_clones': 0, 'num_object_copies': 4328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11325, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11596", 'reported_seq': 11716, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:10.969424+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:10.969424+0000', 'last_peered': '2024-09-11T05:06:10.969424+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:10.969424+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9671840, 'num_objects': 1181, 'num_object_clones': 0, 'num_object_copies': 4724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11594, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1107}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:15.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:16.376 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:16.376 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29972/73932 objects degraded (40.540%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":703,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18483,"data_bytes":151368168,"bytes_used":2827018240,"bytes_avail":383720038400,"bytes_total":386547056640,"degraded_objects":29972,"degraded_total":73932,"degraded_ratio":0.40539955634907754,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":170},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:16.408 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:06:16.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:17.033 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:17.058 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-11T05:06:15.889124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151368168,"num_objects":18483,"num_object_clones":0,"num_object_copies":73932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167420,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2760760,"kb_used_data":298096,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374726600,"statfs":{"total":386547056640,"available":383720038400,"internally_reserved":0,"allocated":305250304,"data_stored":304493496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2521679414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,5,7,4,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16728064,"num_objects":-2042,"num_object_clones":0,"num_object_copies":-8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2042,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005948"},"pg_stats":[{"pgid":"1.f","version":"35'11337","reported_seq":11432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.591246+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:13.591246+0000","last_peered":"2024-09-11T05:06:13.591246+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:13.591246+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9641984,"num_objects":1177,"num_object_clones":0,"num_object_copies":4708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11511","reported_seq":11625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.938623+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:10.938623+0000","last_peered":"2024-09-11T05:06:10.938623+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:10.938623+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10048672,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1031}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11045","reported_seq":11150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.660287+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:13.660287+0000","last_peered":"2024-09-11T05:06:13.660287+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:13.660287+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9132648,"num_objects":1116,"num_object_clones":0,"num_object_copies":4464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11044,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11296","reported_seq":11405,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.949497+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:10.949497+0000","last_peered":"2024-09-11T05:06:10.949497+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:10.949497+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9172128,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11383","reported_seq":11507,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.767907+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:13.767907+0000","last_peered":"2024-09-11T05:06:13.767907+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:13.767907+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9401504,"num_objects":1148,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1073}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11506","reported_seq":11612,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.719315+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:13.719315+0000","last_peered":"2024-09-11T05:06:13.719315+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:13.719315+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9289728,"num_objects":1134,"num_object_clones":0,"num_object_copies":4536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1160}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11515","reported_seq":11609,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.890117+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:10.890117+0000","last_peered":"2024-09-11T05:06:10.890117+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:10.890117+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9950368,"num_objects":1215,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11513,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11675","reported_seq":16533,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:14.182195+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:14.182195+0000","last_peered":"2024-09-11T05:06:14.182195+0000","last_clean":"2024-09-11T05:06:14.182195+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:14.182195+0000","last_undegraded":"2024-09-11T05:06:14.182195+0000","last_fullsized":"2024-09-11T05:06:14.182195+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9404416,"num_objects":1148,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11609","reported_seq":11732,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.712163+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:13.712163+0000","last_peered":"2024-09-11T05:06:13.712163+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:13.712163+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9652544,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11606,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11436","reported_seq":11549,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.655813+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:13.655813+0000","last_peered":"2024-09-11T05:06:13.655813+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:13.655813+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942752,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11434,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11497","reported_seq":11615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.499829+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:13.499829+0000","last_peered":"2024-09-11T05:06:13.499829+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:13.499829+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9465584,"num_objects":1156,"num_object_clones":0,"num_object_copies":4624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11494,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11220","reported_seq":15822,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:14.263149+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:14.263149+0000","last_peered":"2024-09-11T05:06:14.263149+0000","last_clean":"2024-09-11T05:06:14.263149+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:14.263149+0000","last_undegraded":"2024-09-11T05:06:14.263149+0000","last_fullsized":"2024-09-11T05:06:14.263149+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9535488,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11559","reported_seq":15749,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.700355+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:13.700355+0000","last_peered":"2024-09-11T05:06:13.700355+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:13.700355+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9534032,"num_objects":1164,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2053,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11412","reported_seq":11519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.551462+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:13.551462+0000","last_peered":"2024-09-11T05:06:13.551462+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:13.551462+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9663648,"num_objects":1180,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11410,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1084}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11327","reported_seq":11443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:13.493289+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:13.493289+0000","last_peered":"2024-09-11T05:06:13.493289+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:13.493289+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8860832,"num_objects":1082,"num_object_clones":0,"num_object_copies":4328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11596","reported_seq":11716,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:10.969424+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:10.969424+0000","last_peered":"2024-09-11T05:06:10.969424+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:10.969424+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9671840,"num_objects":1181,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":151368168,"num_objects":18483,"num_object_clones":0,"num_object_copies":73932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167420,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302211072,"data_stored":302211072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737168,"kb_used_data":73792,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93634672,"statfs":{"total":96636764160,"available":95881904128,"internally_reserved":0,"allocated":75563008,"data_stored":75373806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715048,"kb_used_data":74384,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93656792,"statfs":{"total":96636764160,"available":95904555008,"internally_reserved":0,"allocated":76169216,"data_stored":75980014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578740,"kb_used_data":73700,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93793100,"statfs":{"total":96636764160,"available":96044134400,"internally_reserved":0,"allocated":75468800,"data_stored":75279598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":517140178},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,2,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729804,"kb_used_data":76220,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93642036,"statfs":{"total":96636764160,"available":95889444864,"internally_reserved":0,"allocated":78049280,"data_stored":77860078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77373440,"data_stored":77373440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74604544,"data_stored":74604544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75116544,"data_stored":75116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75116544,"data_stored":75116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:20.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:20.681 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-11T05:06:19.890811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":143356392,"num_objects":17505,"num_object_clones":0,"num_object_copies":70020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168398,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161002,"ondisk_log_size":161002,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762676,"kb_used_data":281928,"kb_used_omap":14,"kb_used_meta":2480625,"kb_avail":374724684,"statfs":{"total":386547056640,"available":383718076416,"internally_reserved":0,"allocated":288694272,"data_stored":287937464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2540160566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,15,14,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21749760,"num_objects":-2655,"num_object_clones":0,"num_object_copies":-10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005202"},"pg_stats":[{"pgid":"1.f","version":"35'11395","reported_seq":11490,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.659633+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:18.659633+0000","last_peered":"2024-09-11T05:06:18.659633+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:18.659633+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9166848,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11589","reported_seq":11704,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:15.988819+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:15.988819+0000","last_peered":"2024-09-11T05:06:15.988819+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:15.988819+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9409696,"num_objects":1149,"num_object_clones":0,"num_object_copies":4596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11119","reported_seq":11224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.719520+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:18.719520+0000","last_peered":"2024-09-11T05:06:18.719520+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:18.719520+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8526440,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11357","reported_seq":11466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:16.066895+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:16.066895+0000","last_peered":"2024-09-11T05:06:16.066895+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:16.066895+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672416,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11355,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1202}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11455","reported_seq":11579,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.787820+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:18.787820+0000","last_peered":"2024-09-11T05:06:18.787820+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:18.787820+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8811680,"num_objects":1076,"num_object_clones":0,"num_object_copies":4304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11453,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11560","reported_seq":11664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.821557+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:18.821557+0000","last_peered":"2024-09-11T05:06:18.821557+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:18.821557+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8847360,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11589","reported_seq":11683,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:15.977790+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:15.977790+0000","last_peered":"2024-09-11T05:06:15.977790+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:15.977790+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9344160,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11728","reported_seq":16586,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:19.119727+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:19.119727+0000","last_peered":"2024-09-11T05:06:19.119727+0000","last_clean":"2024-09-11T05:06:19.119727+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:19.119727+0000","last_undegraded":"2024-09-11T05:06:19.119727+0000","last_fullsized":"2024-09-11T05:06:19.119727+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8970240,"num_objects":1095,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11660","reported_seq":11784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.342135+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:18.342135+0000","last_peered":"2024-09-11T05:06:18.342135+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:18.342135+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9234752,"num_objects":1128,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11657,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11482","reported_seq":11595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.739289+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:18.739289+0000","last_peered":"2024-09-11T05:06:18.739289+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:18.739289+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8565920,"num_objects":1046,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11480,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11565","reported_seq":11683,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.785188+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:18.785188+0000","last_peered":"2024-09-11T05:06:18.785188+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:18.785188+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8908528,"num_objects":1088,"num_object_clones":0,"num_object_copies":4352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11562,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11277","reported_seq":15879,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:19.473643+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:19.473643+0000","last_peered":"2024-09-11T05:06:19.473643+0000","last_clean":"2024-09-11T05:06:19.473643+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:19.473643+0000","last_undegraded":"2024-09-11T05:06:19.473643+0000","last_fullsized":"2024-09-11T05:06:19.473643+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9068544,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11616","reported_seq":15941,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.892887+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:18.892887+0000","last_peered":"2024-09-11T05:06:18.892887+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:18.892887+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9067088,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11467","reported_seq":11574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.734870+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:18.734870+0000","last_peered":"2024-09-11T05:06:18.734870+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:18.734870+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9213088,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11465,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11386","reported_seq":11506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.866642+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:18.866642+0000","last_peered":"2024-09-11T05:06:18.866642+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:18.866642+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8377504,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11384,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1280}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11657","reported_seq":11775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:16.006784+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:16.006784+0000","last_peered":"2024-09-11T05:06:16.006784+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:16.006784+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9172128,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11655,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":143356392,"num_objects":17505,"num_object_clones":0,"num_object_copies":70020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168398,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286310400,"data_stored":286310400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161002,"ondisk_log_size":161002,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737456,"kb_used_data":69916,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93634384,"statfs":{"total":96636764160,"available":95881609216,"internally_reserved":0,"allocated":71593984,"data_stored":71404782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714600,"kb_used_data":69764,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93657240,"statfs":{"total":96636764160,"available":95905013760,"internally_reserved":0,"allocated":71438336,"data_stored":71249134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597168,"kb_used_data":69920,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93774672,"statfs":{"total":96636764160,"available":96025264128,"internally_reserved":0,"allocated":71598080,"data_stored":71408878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539881170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713452,"kb_used_data":72328,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93658388,"statfs":{"total":96636764160,"available":95906189312,"internally_reserved":0,"allocated":74063872,"data_stored":73874670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656469711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,2,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73379840,"data_stored":73379840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70643712,"data_stored":70643712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71122944,"data_stored":71122944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":71163904,"data_stored":71163904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:20.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:21.304 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:21.305 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:21.322 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-11T05:06:19.890811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":143356392,"num_objects":17505,"num_object_clones":0,"num_object_copies":70020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168398,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161002,"ondisk_log_size":161002,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762676,"kb_used_data":281928,"kb_used_omap":14,"kb_used_meta":2480625,"kb_avail":374724684,"statfs":{"total":386547056640,"available":383718076416,"internally_reserved":0,"allocated":288694272,"data_stored":287937464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2540160566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,15,14,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21749760,"num_objects":-2655,"num_object_clones":0,"num_object_copies":-10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005202"},"pg_stats":[{"pgid":"1.f","version":"35'11395","reported_seq":11490,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.659633+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:18.659633+0000","last_peered":"2024-09-11T05:06:18.659633+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:18.659633+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9166848,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11589","reported_seq":11704,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:15.988819+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:15.988819+0000","last_peered":"2024-09-11T05:06:15.988819+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:15.988819+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9409696,"num_objects":1149,"num_object_clones":0,"num_object_copies":4596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11119","reported_seq":11224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.719520+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:18.719520+0000","last_peered":"2024-09-11T05:06:18.719520+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:18.719520+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8526440,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11357","reported_seq":11466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:16.066895+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:16.066895+0000","last_peered":"2024-09-11T05:06:16.066895+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:16.066895+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672416,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11355,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1202}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11455","reported_seq":11579,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.787820+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:18.787820+0000","last_peered":"2024-09-11T05:06:18.787820+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:18.787820+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8811680,"num_objects":1076,"num_object_clones":0,"num_object_copies":4304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11453,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11560","reported_seq":11664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.821557+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:18.821557+0000","last_peered":"2024-09-11T05:06:18.821557+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:18.821557+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8847360,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11589","reported_seq":11683,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:15.977790+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:15.977790+0000","last_peered":"2024-09-11T05:06:15.977790+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:15.977790+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9344160,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11728","reported_seq":16586,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:19.119727+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:19.119727+0000","last_peered":"2024-09-11T05:06:19.119727+0000","last_clean":"2024-09-11T05:06:19.119727+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:19.119727+0000","last_undegraded":"2024-09-11T05:06:19.119727+0000","last_fullsized":"2024-09-11T05:06:19.119727+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8970240,"num_objects":1095,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11660","reported_seq":11784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.342135+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:18.342135+0000","last_peered":"2024-09-11T05:06:18.342135+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:18.342135+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9234752,"num_objects":1128,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11657,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11482","reported_seq":11595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.739289+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:18.739289+0000","last_peered":"2024-09-11T05:06:18.739289+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:18.739289+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8565920,"num_objects":1046,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11480,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11565","reported_seq":11683,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.785188+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:18.785188+0000","last_peered":"2024-09-11T05:06:18.785188+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:18.785188+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8908528,"num_objects":1088,"num_object_clones":0,"num_object_copies":4352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11562,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11277","reported_seq":15879,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:19.473643+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:19.473643+0000","last_peered":"2024-09-11T05:06:19.473643+0000","last_clean":"2024-09-11T05:06:19.473643+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:19.473643+0000","last_undegraded":"2024-09-11T05:06:19.473643+0000","last_fullsized":"2024-09-11T05:06:19.473643+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9068544,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11616","reported_seq":15941,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.892887+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:18.892887+0000","last_peered":"2024-09-11T05:06:18.892887+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:18.892887+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9067088,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11467","reported_seq":11574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.734870+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:18.734870+0000","last_peered":"2024-09-11T05:06:18.734870+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:18.734870+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9213088,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11465,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11386","reported_seq":11506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.866642+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:18.866642+0000","last_peered":"2024-09-11T05:06:18.866642+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:18.866642+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8377504,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11384,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1280}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11657","reported_seq":11775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:16.006784+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:16.006784+0000","last_peered":"2024-09-11T05:06:16.006784+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:16.006784+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9172128,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11655,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":143356392,"num_objects":17505,"num_object_clones":0,"num_object_copies":70020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168398,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286310400,"data_stored":286310400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161002,"ondisk_log_size":161002,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737456,"kb_used_data":69916,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93634384,"statfs":{"total":96636764160,"available":95881609216,"internally_reserved":0,"allocated":71593984,"data_stored":71404782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714600,"kb_used_data":69764,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93657240,"statfs":{"total":96636764160,"available":95905013760,"internally_reserved":0,"allocated":71438336,"data_stored":71249134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597168,"kb_used_data":69920,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93774672,"statfs":{"total":96636764160,"available":96025264128,"internally_reserved":0,"allocated":71598080,"data_stored":71408878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539881170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713452,"kb_used_data":72328,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93658388,"statfs":{"total":96636764160,"available":95906189312,"internally_reserved":0,"allocated":74063872,"data_stored":73874670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656469711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,2,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73379840,"data_stored":73379840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70643712,"data_stored":70643712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71122944,"data_stored":71122944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":71163904,"data_stored":71163904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:21.323 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:21.323 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11395", 'reported_seq': 11490, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.659633+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:18.659633+0000', 'last_peered': '2024-09-11T05:06:18.659633+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:18.659633+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9166848, 'num_objects': 1119, 'num_object_clones': 0, 'num_object_copies': 4476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11395, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1172}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:21.323 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:21.323 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11589", 'reported_seq': 11704, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:15.988819+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:15.988819+0000', 'last_peered': '2024-09-11T05:06:15.988819+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:15.988819+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9409696, 'num_objects': 1149, 'num_object_clones': 0, 'num_object_copies': 4596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11587, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:21.323 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11119", 'reported_seq': 11224, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.719520+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:18.719520+0000', 'last_peered': '2024-09-11T05:06:18.719520+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:18.719520+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8526440, 'num_objects': 1042, 'num_object_clones': 0, 'num_object_copies': 4168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1042, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11118, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1169}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11357", 'reported_seq': 11466, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:16.066895+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:06:16.066895+0000', 'last_peered': '2024-09-11T05:06:16.066895+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:16.066895+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8672416, 'num_objects': 1059, 'num_object_clones': 0, 'num_object_copies': 4236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11355, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11455", 'reported_seq': 11579, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.787820+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:18.787820+0000', 'last_peered': '2024-09-11T05:06:18.787820+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:18.787820+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8811680, 'num_objects': 1076, 'num_object_clones': 0, 'num_object_copies': 4304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11453, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1145}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11560", 'reported_seq': 11664, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.821557+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:18.821557+0000', 'last_peered': '2024-09-11T05:06:18.821557+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:18.821557+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8847360, 'num_objects': 1080, 'num_object_clones': 0, 'num_object_copies': 4320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1214}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11589", 'reported_seq': 11683, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:15.977790+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:15.977790+0000', 'last_peered': '2024-09-11T05:06:15.977790+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:15.977790+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9344160, 'num_objects': 1141, 'num_object_clones': 0, 'num_object_copies': 4564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11587, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1166}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:21.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11660", 'reported_seq': 11784, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.342135+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:18.342135+0000', 'last_peered': '2024-09-11T05:06:18.342135+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:18.342135+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9234752, 'num_objects': 1128, 'num_object_clones': 0, 'num_object_copies': 4512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11657, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1238}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11482", 'reported_seq': 11595, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.739289+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:18.739289+0000', 'last_peered': '2024-09-11T05:06:18.739289+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:18.739289+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8565920, 'num_objects': 1046, 'num_object_clones': 0, 'num_object_copies': 4184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11480, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1260}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11565", 'reported_seq': 11683, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.785188+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:18.785188+0000', 'last_peered': '2024-09-11T05:06:18.785188+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:18.785188+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8908528, 'num_objects': 1088, 'num_object_clones': 0, 'num_object_copies': 4352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11562, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1210}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11616", 'reported_seq': 15941, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.892887+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:06:18.892887+0000', 'last_peered': '2024-09-11T05:06:18.892887+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:06:18.892887+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9067088, 'num_objects': 1107, 'num_object_clones': 0, 'num_object_copies': 4428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11615, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2120, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11467", 'reported_seq': 11574, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.734870+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:18.734870+0000', 'last_peered': '2024-09-11T05:06:18.734870+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:18.734870+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9213088, 'num_objects': 1125, 'num_object_clones': 0, 'num_object_copies': 4500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11465, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1139}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:21.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:21.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11386", 'reported_seq': 11506, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:18.866642+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:18.866642+0000', 'last_peered': '2024-09-11T05:06:18.866642+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:18.866642+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8377504, 'num_objects': 1023, 'num_object_clones': 0, 'num_object_copies': 4092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1023, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11384, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:21.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:21.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11657", 'reported_seq': 11775, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:16.006784+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:16.006784+0000', 'last_peered': '2024-09-11T05:06:16.006784+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:16.006784+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9172128, 'num_objects': 1120, 'num_object_clones': 0, 'num_object_copies': 4480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11655, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:21.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:22.089 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:22.089 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29905/70020 objects degraded (42.709%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":709,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":17505,"data_bytes":143356392,"bytes_used":2828980224,"bytes_avail":383718076416,"bytes_total":386547056640,"degraded_objects":29905,"degraded_total":70020,"degraded_ratio":0.42709225935447015,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:22.114 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:06:22.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:22.779 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:22.807 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-11T05:06:21.891656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":141275624,"num_objects":17251,"num_object_clones":0,"num_object_copies":69004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168652,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762932,"kb_used_data":278040,"kb_used_omap":14,"kb_used_meta":2484785,"kb_avail":374724428,"statfs":{"total":386547056640,"available":383717814272,"internally_reserved":0,"allocated":284712960,"data_stored":283956152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2544420406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,10,18,14,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18022400,"num_objects":-2200,"num_object_clones":0,"num_object_copies":-8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005205"},"pg_stats":[{"pgid":"1.f","version":"35'11395","reported_seq":11490,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.659633+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:18.659633+0000","last_peered":"2024-09-11T05:06:18.659633+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:18.659633+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9166848,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11658","reported_seq":11769,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:21.044006+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:21.044006+0000","last_peered":"2024-09-11T05:06:21.044006+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:21.044006+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844448,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11119","reported_seq":11224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.719520+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:18.719520+0000","last_peered":"2024-09-11T05:06:18.719520+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:18.719520+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8526440,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11418","reported_seq":11527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:20.861512+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:20.861512+0000","last_peered":"2024-09-11T05:06:20.861512+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:20.861512+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8172704,"num_objects":998,"num_object_clones":0,"num_object_copies":3992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11455","reported_seq":11579,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.787820+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:18.787820+0000","last_peered":"2024-09-11T05:06:18.787820+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:18.787820+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8811680,"num_objects":1076,"num_object_clones":0,"num_object_copies":4304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11453,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11560","reported_seq":11664,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.821557+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:18.821557+0000","last_peered":"2024-09-11T05:06:18.821557+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:18.821557+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8847360,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11656","reported_seq":11751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:21.070871+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:21.070871+0000","last_peered":"2024-09-11T05:06:21.070871+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:21.070871+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8795296,"num_objects":1074,"num_object_clones":0,"num_object_copies":4296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11654,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11728","reported_seq":16586,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:19.119727+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:19.119727+0000","last_peered":"2024-09-11T05:06:19.119727+0000","last_clean":"2024-09-11T05:06:19.119727+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:19.119727+0000","last_undegraded":"2024-09-11T05:06:19.119727+0000","last_fullsized":"2024-09-11T05:06:19.119727+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8970240,"num_objects":1095,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11660","reported_seq":11784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.342135+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:18.342135+0000","last_peered":"2024-09-11T05:06:18.342135+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:18.342135+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9234752,"num_objects":1128,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11657,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11482","reported_seq":11595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.739289+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:18.739289+0000","last_peered":"2024-09-11T05:06:18.739289+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:18.739289+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8565920,"num_objects":1046,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11480,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11565","reported_seq":11683,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.785188+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:18.785188+0000","last_peered":"2024-09-11T05:06:18.785188+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:18.785188+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8908528,"num_objects":1088,"num_object_clones":0,"num_object_copies":4352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11562,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11277","reported_seq":15879,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:19.473643+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:19.473643+0000","last_peered":"2024-09-11T05:06:19.473643+0000","last_clean":"2024-09-11T05:06:19.473643+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:19.473643+0000","last_undegraded":"2024-09-11T05:06:19.473643+0000","last_fullsized":"2024-09-11T05:06:19.473643+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9068544,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11616","reported_seq":15941,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.892887+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:18.892887+0000","last_peered":"2024-09-11T05:06:18.892887+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:18.892887+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9067088,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11467","reported_seq":11574,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.734870+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:18.734870+0000","last_peered":"2024-09-11T05:06:18.734870+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:18.734870+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9213088,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11465,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11386","reported_seq":11506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:18.866642+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:18.866642+0000","last_peered":"2024-09-11T05:06:18.866642+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:18.866642+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8377504,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11384,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1280}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11714","reported_seq":11834,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:20.905048+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:20.905048+0000","last_peered":"2024-09-11T05:06:20.905048+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:20.905048+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8705184,"num_objects":1063,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":141275624,"num_objects":17251,"num_object_clones":0,"num_object_copies":69004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168652,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282312704,"data_stored":282312704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737456,"kb_used_data":69916,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93634384,"statfs":{"total":96636764160,"available":95881609216,"internally_reserved":0,"allocated":71593984,"data_stored":71404782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714600,"kb_used_data":69764,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93657240,"statfs":{"total":96636764160,"available":95905013760,"internally_reserved":0,"allocated":71438336,"data_stored":71249134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597168,"kb_used_data":69920,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93774672,"statfs":{"total":96636764160,"available":96025264128,"internally_reserved":0,"allocated":71598080,"data_stored":71408878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":539881170},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":68440,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":70082560,"data_stored":69893358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69382144,"data_stored":69382144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70643712,"data_stored":70643712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71122944,"data_stored":71122944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":71163904,"data_stored":71163904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:25.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:26.506 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:26.548 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-11T05:06:25.893327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135475688,"num_objects":16543,"num_object_clones":0,"num_object_copies":66172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169360,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160664,"ondisk_log_size":160664,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747348,"kb_used_data":266628,"kb_used_omap":14,"kb_used_meta":2480625,"kb_avail":374740012,"statfs":{"total":386547056640,"available":383733772288,"internally_reserved":0,"allocated":273027072,"data_stored":272270264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2540160566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,8,5,19,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17121280,"num_objects":-2090,"num_object_clones":0,"num_object_copies":-8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005081"},"pg_stats":[{"pgid":"1.f","version":"35'11455","reported_seq":11550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.843198+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:23.843198+0000","last_peered":"2024-09-11T05:06:23.843198+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:23.843198+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8675328,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11455,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11658","reported_seq":11769,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:21.044006+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:21.044006+0000","last_peered":"2024-09-11T05:06:21.044006+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:21.044006+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844448,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11186","reported_seq":11292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.739839+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:23.739839+0000","last_peered":"2024-09-11T05:06:23.739839+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:23.739839+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7977576,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11185,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11418","reported_seq":11527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:20.861512+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:20.861512+0000","last_peered":"2024-09-11T05:06:20.861512+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:20.861512+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8172704,"num_objects":998,"num_object_clones":0,"num_object_copies":3992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11509","reported_seq":11633,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.763473+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:23.763473+0000","last_peered":"2024-09-11T05:06:23.763473+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:23.763473+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8369312,"num_objects":1022,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11613","reported_seq":11718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.848108+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:23.848108+0000","last_peered":"2024-09-11T05:06:23.848108+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:23.848108+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8413184,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11613,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11656","reported_seq":11751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:21.070871+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:21.070871+0000","last_peered":"2024-09-11T05:06:21.070871+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:21.070871+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8795296,"num_objects":1074,"num_object_clones":0,"num_object_copies":4296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11654,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11783","reported_seq":16641,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:24.574349+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:24.574349+0000","last_peered":"2024-09-11T05:06:24.574349+0000","last_clean":"2024-09-11T05:06:24.574349+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:24.574349+0000","last_undegraded":"2024-09-11T05:06:24.574349+0000","last_fullsized":"2024-09-11T05:06:24.574349+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8519680,"num_objects":1040,"num_object_clones":0,"num_object_copies":4160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11723","reported_seq":11844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.794875+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:23.794875+0000","last_peered":"2024-09-11T05:06:23.794875+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:23.794875+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8718656,"num_objects":1065,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11544","reported_seq":11657,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.752644+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:23.752644+0000","last_peered":"2024-09-11T05:06:23.752644+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:23.752644+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8058016,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11542,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11626","reported_seq":11743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.728727+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:23.728727+0000","last_peered":"2024-09-11T05:06:23.728727+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:23.728727+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8408816,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11623,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11352","reported_seq":15953,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:24.524451+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:24.524451+0000","last_peered":"2024-09-11T05:06:24.524451+0000","last_clean":"2024-09-11T05:06:24.524451+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:24.524451+0000","last_undegraded":"2024-09-11T05:06:24.524451+0000","last_fullsized":"2024-09-11T05:06:24.524451+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5162,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11662","reported_seq":16107,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.964224+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:23.964224+0000","last_peered":"2024-09-11T05:06:23.964224+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:23.964224+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8690256,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11661,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11524","reported_seq":11631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.841467+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:23.841467+0000","last_peered":"2024-09-11T05:06:23.841467+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:23.841467+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746144,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11441","reported_seq":11561,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.773791+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:23.773791+0000","last_peered":"2024-09-11T05:06:23.773791+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:23.773791+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7926944,"num_objects":968,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11714","reported_seq":11834,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:20.905048+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:20.905048+0000","last_peered":"2024-09-11T05:06:20.905048+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:20.905048+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8705184,"num_objects":1063,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135475688,"num_objects":16543,"num_object_clones":0,"num_object_copies":66172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169360,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270536704,"data_stored":270536704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160664,"ondisk_log_size":160664,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737816,"kb_used_data":66120,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93634024,"statfs":{"total":96636764160,"available":95881240576,"internally_reserved":0,"allocated":67706880,"data_stored":67517678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,4,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714952,"kb_used_data":65964,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93656888,"statfs":{"total":96636764160,"available":95904653312,"internally_reserved":0,"allocated":67547136,"data_stored":67357934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580872,"kb_used_data":66104,"kb_used_omap":4,"kb_used_meta":514747,"kb_avail":93790968,"statfs":{"total":96636764160,"available":96041951232,"internally_reserved":0,"allocated":67690496,"data_stored":67501294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527101650},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":68440,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":70082560,"data_stored":69893358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69382144,"data_stored":69382144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66736128,"data_stored":66736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67166208,"data_stored":67166208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67252224,"data_stored":67252224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:26.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:27.206 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:27.228 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-11T05:06:25.893327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135475688,"num_objects":16543,"num_object_clones":0,"num_object_copies":66172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169360,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160664,"ondisk_log_size":160664,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747348,"kb_used_data":266628,"kb_used_omap":14,"kb_used_meta":2480625,"kb_avail":374740012,"statfs":{"total":386547056640,"available":383733772288,"internally_reserved":0,"allocated":273027072,"data_stored":272270264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2540160566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,8,5,19,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17121280,"num_objects":-2090,"num_object_clones":0,"num_object_copies":-8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005081"},"pg_stats":[{"pgid":"1.f","version":"35'11455","reported_seq":11550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.843198+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:23.843198+0000","last_peered":"2024-09-11T05:06:23.843198+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:23.843198+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8675328,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11455,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11658","reported_seq":11769,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:21.044006+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:21.044006+0000","last_peered":"2024-09-11T05:06:21.044006+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:21.044006+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844448,"num_objects":1080,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11186","reported_seq":11292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.739839+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:23.739839+0000","last_peered":"2024-09-11T05:06:23.739839+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:23.739839+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7977576,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11185,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11418","reported_seq":11527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:20.861512+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:20.861512+0000","last_peered":"2024-09-11T05:06:20.861512+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:20.861512+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8172704,"num_objects":998,"num_object_clones":0,"num_object_copies":3992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11509","reported_seq":11633,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.763473+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:23.763473+0000","last_peered":"2024-09-11T05:06:23.763473+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:23.763473+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8369312,"num_objects":1022,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11613","reported_seq":11718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.848108+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:23.848108+0000","last_peered":"2024-09-11T05:06:23.848108+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:23.848108+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8413184,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11613,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11656","reported_seq":11751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:21.070871+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:21.070871+0000","last_peered":"2024-09-11T05:06:21.070871+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:21.070871+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8795296,"num_objects":1074,"num_object_clones":0,"num_object_copies":4296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11654,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11783","reported_seq":16641,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:24.574349+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:24.574349+0000","last_peered":"2024-09-11T05:06:24.574349+0000","last_clean":"2024-09-11T05:06:24.574349+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:24.574349+0000","last_undegraded":"2024-09-11T05:06:24.574349+0000","last_fullsized":"2024-09-11T05:06:24.574349+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8519680,"num_objects":1040,"num_object_clones":0,"num_object_copies":4160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11723","reported_seq":11844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.794875+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:23.794875+0000","last_peered":"2024-09-11T05:06:23.794875+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:23.794875+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8718656,"num_objects":1065,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11544","reported_seq":11657,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.752644+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:23.752644+0000","last_peered":"2024-09-11T05:06:23.752644+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:23.752644+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8058016,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11542,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11626","reported_seq":11743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.728727+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:23.728727+0000","last_peered":"2024-09-11T05:06:23.728727+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:23.728727+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8408816,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11623,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11352","reported_seq":15953,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:24.524451+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:24.524451+0000","last_peered":"2024-09-11T05:06:24.524451+0000","last_clean":"2024-09-11T05:06:24.524451+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:24.524451+0000","last_undegraded":"2024-09-11T05:06:24.524451+0000","last_fullsized":"2024-09-11T05:06:24.524451+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5162,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11662","reported_seq":16107,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.964224+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:23.964224+0000","last_peered":"2024-09-11T05:06:23.964224+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:23.964224+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8690256,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11661,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11524","reported_seq":11631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.841467+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:23.841467+0000","last_peered":"2024-09-11T05:06:23.841467+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:23.841467+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746144,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11441","reported_seq":11561,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.773791+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:23.773791+0000","last_peered":"2024-09-11T05:06:23.773791+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:23.773791+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7926944,"num_objects":968,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11714","reported_seq":11834,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:20.905048+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:20.905048+0000","last_peered":"2024-09-11T05:06:20.905048+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:20.905048+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8705184,"num_objects":1063,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135475688,"num_objects":16543,"num_object_clones":0,"num_object_copies":66172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169360,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270536704,"data_stored":270536704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160664,"ondisk_log_size":160664,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737816,"kb_used_data":66120,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93634024,"statfs":{"total":96636764160,"available":95881240576,"internally_reserved":0,"allocated":67706880,"data_stored":67517678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,4,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714952,"kb_used_data":65964,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93656888,"statfs":{"total":96636764160,"available":95904653312,"internally_reserved":0,"allocated":67547136,"data_stored":67357934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580872,"kb_used_data":66104,"kb_used_omap":4,"kb_used_meta":514747,"kb_avail":93790968,"statfs":{"total":96636764160,"available":96041951232,"internally_reserved":0,"allocated":67690496,"data_stored":67501294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527101650},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":68440,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":70082560,"data_stored":69893358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69382144,"data_stored":69382144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66736128,"data_stored":66736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67166208,"data_stored":67166208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67252224,"data_stored":67252224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:27.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:27.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11455", 'reported_seq': 11550, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.843198+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:23.843198+0000', 'last_peered': '2024-09-11T05:06:23.843198+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:23.843198+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8675328, 'num_objects': 1059, 'num_object_clones': 0, 'num_object_copies': 4236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11455, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1232}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:27.230 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:27.230 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11658", 'reported_seq': 11769, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:21.044006+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:21.044006+0000', 'last_peered': '2024-09-11T05:06:21.044006+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:21.044006+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8844448, 'num_objects': 1080, 'num_object_clones': 0, 'num_object_copies': 4320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11656, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:27.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:27.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11186", 'reported_seq': 11292, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.739839+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:23.739839+0000', 'last_peered': '2024-09-11T05:06:23.739839+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:23.739839+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7977576, 'num_objects': 975, 'num_object_clones': 0, 'num_object_copies': 3900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 975, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11185, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1236}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:27.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:27.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11418", 'reported_seq': 11527, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:20.861512+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:06:20.861512+0000', 'last_peered': '2024-09-11T05:06:20.861512+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:20.861512+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8172704, 'num_objects': 998, 'num_object_clones': 0, 'num_object_copies': 3992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 998, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11416, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:27.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:27.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11509", 'reported_seq': 11633, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.763473+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:23.763473+0000', 'last_peered': '2024-09-11T05:06:23.763473+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:23.763473+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8369312, 'num_objects': 1022, 'num_object_clones': 0, 'num_object_copies': 4088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1022, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11507, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1199}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:27.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:27.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11613", 'reported_seq': 11718, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.848108+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:23.848108+0000', 'last_peered': '2024-09-11T05:06:23.848108+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:23.848108+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8413184, 'num_objects': 1027, 'num_object_clones': 0, 'num_object_copies': 4108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11613, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1267}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:27.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:27.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11656", 'reported_seq': 11751, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:21.070871+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:21.070871+0000', 'last_peered': '2024-09-11T05:06:21.070871+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:21.070871+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8795296, 'num_objects': 1074, 'num_object_clones': 0, 'num_object_copies': 4296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11654, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1233}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:27.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:27.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11723", 'reported_seq': 11844, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.794875+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:23.794875+0000', 'last_peered': '2024-09-11T05:06:23.794875+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:23.794875+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8718656, 'num_objects': 1065, 'num_object_clones': 0, 'num_object_copies': 4260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11720, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1301}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:27.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:27.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11544", 'reported_seq': 11657, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.752644+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:23.752644+0000', 'last_peered': '2024-09-11T05:06:23.752644+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:23.752644+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8058016, 'num_objects': 984, 'num_object_clones': 0, 'num_object_copies': 3936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11542, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1322}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:27.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:27.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11626", 'reported_seq': 11743, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.728727+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:23.728727+0000', 'last_peered': '2024-09-11T05:06:23.728727+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:23.728727+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8408816, 'num_objects': 1027, 'num_object_clones': 0, 'num_object_copies': 4108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11623, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1271}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:27.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:06:27.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11662", 'reported_seq': 16107, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.964224+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:06:23.964224+0000', 'last_peered': '2024-09-11T05:06:23.964224+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:06:23.964224+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8690256, 'num_objects': 1061, 'num_object_clones': 0, 'num_object_copies': 4244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 154, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11661, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2180, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:27.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:27.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11524", 'reported_seq': 11631, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.841467+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:23.841467+0000', 'last_peered': '2024-09-11T05:06:23.841467+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:23.841467+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8746144, 'num_objects': 1068, 'num_object_clones': 0, 'num_object_copies': 4272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11522, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:27.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:27.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11441", 'reported_seq': 11561, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:23.773791+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:23.773791+0000', 'last_peered': '2024-09-11T05:06:23.773791+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:23.773791+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7926944, 'num_objects': 968, 'num_object_clones': 0, 'num_object_copies': 3872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11439, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:27.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:27.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11714", 'reported_seq': 11834, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:20.905048+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:20.905048+0000', 'last_peered': '2024-09-11T05:06:20.905048+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:20.905048+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8705184, 'num_objects': 1063, 'num_object_clones': 0, 'num_object_copies': 4252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11712, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1225}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:27.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:28.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:28.009 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29845/65108 objects degraded (45.839%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":715,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16277,"data_bytes":133296616,"bytes_used":2813665280,"bytes_avail":383733391360,"bytes_total":386547056640,"degraded_objects":29845,"degraded_total":65108,"degraded_ratio":0.45839220986668311,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":183},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:28.030 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:06:28.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:28.642 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:28.642 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:28.672 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-11T05:06:27.894271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":133296616,"num_objects":16277,"num_object_clones":0,"num_object_copies":65108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169626,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160730,"ondisk_log_size":160730,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747720,"kb_used_data":262844,"kb_used_omap":14,"kb_used_meta":2484785,"kb_avail":374739640,"statfs":{"total":386547056640,"available":383733391360,"internally_reserved":0,"allocated":269152256,"data_stored":268395448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2544420406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,5,11,19,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18071552,"num_objects":-2206,"num_object_clones":0,"num_object_copies":-8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2206,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005146"},"pg_stats":[{"pgid":"1.f","version":"35'11455","reported_seq":11550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.843198+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:23.843198+0000","last_peered":"2024-09-11T05:06:23.843198+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:23.843198+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8675328,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11455,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11729","reported_seq":11844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:26.109751+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:26.109751+0000","last_peered":"2024-09-11T05:06:26.109751+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:26.109751+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8262816,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11186","reported_seq":11292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.739839+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:23.739839+0000","last_peered":"2024-09-11T05:06:23.739839+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:23.739839+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7977576,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11185,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11489","reported_seq":11598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:25.839148+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:25.839148+0000","last_peered":"2024-09-11T05:06:25.839148+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:25.839148+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7591072,"num_objects":927,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11487,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11509","reported_seq":11633,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.763473+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:23.763473+0000","last_peered":"2024-09-11T05:06:23.763473+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:23.763473+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8369312,"num_objects":1022,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11613","reported_seq":11718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.848108+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:23.848108+0000","last_peered":"2024-09-11T05:06:23.848108+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:23.848108+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8413184,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11613,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11704","reported_seq":11799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:26.046805+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:26.046805+0000","last_peered":"2024-09-11T05:06:26.046805+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:26.046805+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8402080,"num_objects":1026,"num_object_clones":0,"num_object_copies":4104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1281}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11783","reported_seq":16641,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:24.574349+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:24.574349+0000","last_peered":"2024-09-11T05:06:24.574349+0000","last_clean":"2024-09-11T05:06:24.574349+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:24.574349+0000","last_undegraded":"2024-09-11T05:06:24.574349+0000","last_fullsized":"2024-09-11T05:06:24.574349+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8519680,"num_objects":1040,"num_object_clones":0,"num_object_copies":4160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11723","reported_seq":11844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.794875+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:23.794875+0000","last_peered":"2024-09-11T05:06:23.794875+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:23.794875+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8718656,"num_objects":1065,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11544","reported_seq":11657,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.752644+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:23.752644+0000","last_peered":"2024-09-11T05:06:23.752644+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:23.752644+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8058016,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11542,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11626","reported_seq":11743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.728727+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:23.728727+0000","last_peered":"2024-09-11T05:06:23.728727+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:23.728727+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8408816,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11623,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11352","reported_seq":15953,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:24.524451+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:24.524451+0000","last_peered":"2024-09-11T05:06:24.524451+0000","last_clean":"2024-09-11T05:06:24.524451+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:24.524451+0000","last_undegraded":"2024-09-11T05:06:24.524451+0000","last_fullsized":"2024-09-11T05:06:24.524451+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5162,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11662","reported_seq":16107,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.964224+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:23.964224+0000","last_peered":"2024-09-11T05:06:23.964224+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:23.964224+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8690256,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11661,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11524","reported_seq":11631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.841467+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:23.841467+0000","last_peered":"2024-09-11T05:06:23.841467+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:23.841467+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746144,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11441","reported_seq":11561,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:23.773791+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:23.773791+0000","last_peered":"2024-09-11T05:06:23.773791+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:23.773791+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7926944,"num_objects":968,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11790","reported_seq":11909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:26.102518+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:26.102518+0000","last_peered":"2024-09-11T05:06:26.102518+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:26.102518+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8082592,"num_objects":987,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":133296616,"num_objects":16277,"num_object_clones":0,"num_object_copies":65108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169626,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2463,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266543104,"data_stored":266543104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160730,"ondisk_log_size":160730,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737816,"kb_used_data":66120,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93634024,"statfs":{"total":96636764160,"available":95881240576,"internally_reserved":0,"allocated":67706880,"data_stored":67517678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,4,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714952,"kb_used_data":65964,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93656888,"statfs":{"total":96636764160,"available":95904653312,"internally_reserved":0,"allocated":67547136,"data_stored":67357934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580872,"kb_used_data":66104,"kb_used_omap":4,"kb_used_meta":514747,"kb_avail":93790968,"statfs":{"total":96636764160,"available":96041951232,"internally_reserved":0,"allocated":67690496,"data_stored":67501294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":527101650},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714080,"kb_used_data":64656,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93657760,"statfs":{"total":96636764160,"available":95905546240,"internally_reserved":0,"allocated":66207744,"data_stored":66018542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,6,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65388544,"data_stored":65388544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66736128,"data_stored":66736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67166208,"data_stored":67166208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67252224,"data_stored":67252224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:31.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:32.331 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:32.379 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-11T05:06:31.895990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125596136,"num_objects":15337,"num_object_clones":0,"num_object_copies":61348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170566,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2525,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160970,"ondisk_log_size":160970,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767176,"kb_used_data":247588,"kb_used_omap":14,"kb_used_meta":2519473,"kb_avail":374720184,"statfs":{"total":386547056640,"available":383713468416,"internally_reserved":0,"allocated":253530112,"data_stored":252773304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2579940918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,4,11,7,17,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17760256,"num_objects":-2168,"num_object_clones":0,"num_object_copies":-8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005179"},"pg_stats":[{"pgid":"1.f","version":"35'11525","reported_seq":11620,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.864814+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:28.864814+0000","last_peered":"2024-09-11T05:06:28.864814+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:28.864814+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101888,"num_objects":989,"num_object_clones":0,"num_object_copies":3956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11525,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11798","reported_seq":11913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.210036+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:31.210036+0000","last_peered":"2024-09-11T05:06:31.210036+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:31.210036+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7697568,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11796,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11237","reported_seq":11343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.746217+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:28.746217+0000","last_peered":"2024-09-11T05:06:28.746217+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:28.746217+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7559784,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11236,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11541","reported_seq":11650,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.120171+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:31.120171+0000","last_peered":"2024-09-11T05:06:31.120171+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:31.120171+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7165088,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11576","reported_seq":11700,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.923741+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:28.923741+0000","last_peered":"2024-09-11T05:06:28.923741+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:28.923741+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820448,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11663","reported_seq":11769,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.705940+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:28.705940+0000","last_peered":"2024-09-11T05:06:28.705940+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:28.705940+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8003584,"num_objects":977,"num_object_clones":0,"num_object_copies":3908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11663,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11767","reported_seq":11862,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.261337+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:31.261337+0000","last_peered":"2024-09-11T05:06:31.261337+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:31.261337+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7885984,"num_objects":963,"num_object_clones":0,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11765,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1344}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11843","reported_seq":16701,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:29.678930+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:29.678930+0000","last_peered":"2024-09-11T05:06:29.678930+0000","last_clean":"2024-09-11T05:06:29.678930+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:29.678930+0000","last_undegraded":"2024-09-11T05:06:29.678930+0000","last_fullsized":"2024-09-11T05:06:29.678930+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8028160,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11781","reported_seq":11905,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.784422+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:28.784422+0000","last_peered":"2024-09-11T05:06:28.784422+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:28.784422+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8243520,"num_objects":1007,"num_object_clones":0,"num_object_copies":4028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11778,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11592","reported_seq":11705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.914204+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:28.914204+0000","last_peered":"2024-09-11T05:06:28.914204+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:28.914204+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7664800,"num_objects":936,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11681","reported_seq":11799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.889320+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:28.889320+0000","last_peered":"2024-09-11T05:06:28.889320+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:28.889320+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7958256,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11405","reported_seq":16007,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:29.678949+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:29.678949+0000","last_peered":"2024-09-11T05:06:29.678949+0000","last_clean":"2024-09-11T05:06:29.678949+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:29.678949+0000","last_undegraded":"2024-09-11T05:06:29.678949+0000","last_fullsized":"2024-09-11T05:06:29.678949+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8019968,"num_objects":979,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11721","reported_seq":16290,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.989785+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:28.989785+0000","last_peered":"2024-09-11T05:06:28.989785+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:28.989785+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8206928,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11599","reported_seq":11705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.870110+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:28.870110+0000","last_peered":"2024-09-11T05:06:28.870110+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:28.870110+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8131744,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11597,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1271}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11495","reported_seq":11615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.706788+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:28.706788+0000","last_peered":"2024-09-11T05:06:28.706788+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:28.706788+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7484576,"num_objects":914,"num_object_clones":0,"num_object_copies":3656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11846","reported_seq":11966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.259859+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:31.259859+0000","last_peered":"2024-09-11T05:06:31.259859+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:31.259859+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7623840,"num_objects":931,"num_object_clones":0,"num_object_copies":3724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":125596136,"num_objects":15337,"num_object_clones":0,"num_object_copies":61348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170566,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2525,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250724352,"data_stored":250724352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160970,"ondisk_log_size":160970,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738164,"kb_used_data":62308,"kb_used_omap":4,"kb_used_meta":675835,"kb_avail":93633676,"statfs":{"total":96636764160,"available":95880884224,"internally_reserved":0,"allocated":63803392,"data_stored":63614190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":692055760},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,7,0,4,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715320,"kb_used_data":62180,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93656520,"statfs":{"total":96636764160,"available":95904276480,"internally_reserved":0,"allocated":63672320,"data_stored":63483118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599264,"kb_used_data":62276,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93772576,"statfs":{"total":96636764160,"available":96023117824,"internally_reserved":0,"allocated":63770624,"data_stored":63581422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549842642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714428,"kb_used_data":60824,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93657412,"statfs":{"total":96636764160,"available":95905189888,"internally_reserved":0,"allocated":62283776,"data_stored":62094574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61370368,"data_stored":61370368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62754816,"data_stored":62754816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63275008,"data_stored":63275008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63324160,"data_stored":63324160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:32.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:33.070 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:33.070 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:33.095 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-11T05:06:31.895990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125596136,"num_objects":15337,"num_object_clones":0,"num_object_copies":61348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170566,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2525,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160970,"ondisk_log_size":160970,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767176,"kb_used_data":247588,"kb_used_omap":14,"kb_used_meta":2519473,"kb_avail":374720184,"statfs":{"total":386547056640,"available":383713468416,"internally_reserved":0,"allocated":253530112,"data_stored":252773304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2579940918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,4,11,7,17,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17760256,"num_objects":-2168,"num_object_clones":0,"num_object_copies":-8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005179"},"pg_stats":[{"pgid":"1.f","version":"35'11525","reported_seq":11620,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.864814+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:28.864814+0000","last_peered":"2024-09-11T05:06:28.864814+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:28.864814+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101888,"num_objects":989,"num_object_clones":0,"num_object_copies":3956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11525,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11798","reported_seq":11913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.210036+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:31.210036+0000","last_peered":"2024-09-11T05:06:31.210036+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:31.210036+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7697568,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11796,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11237","reported_seq":11343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.746217+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:28.746217+0000","last_peered":"2024-09-11T05:06:28.746217+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:28.746217+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7559784,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11236,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11541","reported_seq":11650,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.120171+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:31.120171+0000","last_peered":"2024-09-11T05:06:31.120171+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:31.120171+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7165088,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11576","reported_seq":11700,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.923741+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:28.923741+0000","last_peered":"2024-09-11T05:06:28.923741+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:28.923741+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820448,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11663","reported_seq":11769,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.705940+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:28.705940+0000","last_peered":"2024-09-11T05:06:28.705940+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:28.705940+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8003584,"num_objects":977,"num_object_clones":0,"num_object_copies":3908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11663,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11767","reported_seq":11862,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.261337+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:31.261337+0000","last_peered":"2024-09-11T05:06:31.261337+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:31.261337+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7885984,"num_objects":963,"num_object_clones":0,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11765,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1344}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11843","reported_seq":16701,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:29.678930+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:29.678930+0000","last_peered":"2024-09-11T05:06:29.678930+0000","last_clean":"2024-09-11T05:06:29.678930+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:29.678930+0000","last_undegraded":"2024-09-11T05:06:29.678930+0000","last_fullsized":"2024-09-11T05:06:29.678930+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8028160,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11781","reported_seq":11905,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.784422+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:28.784422+0000","last_peered":"2024-09-11T05:06:28.784422+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:28.784422+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8243520,"num_objects":1007,"num_object_clones":0,"num_object_copies":4028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11778,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11592","reported_seq":11705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.914204+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:28.914204+0000","last_peered":"2024-09-11T05:06:28.914204+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:28.914204+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7664800,"num_objects":936,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11681","reported_seq":11799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.889320+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:28.889320+0000","last_peered":"2024-09-11T05:06:28.889320+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:28.889320+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7958256,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11405","reported_seq":16007,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:29.678949+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:29.678949+0000","last_peered":"2024-09-11T05:06:29.678949+0000","last_clean":"2024-09-11T05:06:29.678949+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:29.678949+0000","last_undegraded":"2024-09-11T05:06:29.678949+0000","last_fullsized":"2024-09-11T05:06:29.678949+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8019968,"num_objects":979,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11721","reported_seq":16290,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.989785+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:28.989785+0000","last_peered":"2024-09-11T05:06:28.989785+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:28.989785+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8206928,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11599","reported_seq":11705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.870110+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:28.870110+0000","last_peered":"2024-09-11T05:06:28.870110+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:28.870110+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8131744,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11597,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1271}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11495","reported_seq":11615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.706788+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:28.706788+0000","last_peered":"2024-09-11T05:06:28.706788+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:28.706788+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7484576,"num_objects":914,"num_object_clones":0,"num_object_copies":3656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11846","reported_seq":11966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.259859+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:31.259859+0000","last_peered":"2024-09-11T05:06:31.259859+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:31.259859+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7623840,"num_objects":931,"num_object_clones":0,"num_object_copies":3724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":125596136,"num_objects":15337,"num_object_clones":0,"num_object_copies":61348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170566,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2525,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250724352,"data_stored":250724352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160970,"ondisk_log_size":160970,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738164,"kb_used_data":62308,"kb_used_omap":4,"kb_used_meta":675835,"kb_avail":93633676,"statfs":{"total":96636764160,"available":95880884224,"internally_reserved":0,"allocated":63803392,"data_stored":63614190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":692055760},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,7,0,4,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715320,"kb_used_data":62180,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93656520,"statfs":{"total":96636764160,"available":95904276480,"internally_reserved":0,"allocated":63672320,"data_stored":63483118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599264,"kb_used_data":62276,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93772576,"statfs":{"total":96636764160,"available":96023117824,"internally_reserved":0,"allocated":63770624,"data_stored":63581422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549842642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714428,"kb_used_data":60824,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93657412,"statfs":{"total":96636764160,"available":95905189888,"internally_reserved":0,"allocated":62283776,"data_stored":62094574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61370368,"data_stored":61370368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62754816,"data_stored":62754816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63275008,"data_stored":63275008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63324160,"data_stored":63324160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:33.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:33.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11525", 'reported_seq': 11620, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.864814+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:28.864814+0000', 'last_peered': '2024-09-11T05:06:28.864814+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:28.864814+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8101888, 'num_objects': 989, 'num_object_clones': 0, 'num_object_copies': 3956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11525, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1302}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:33.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:33.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11798", 'reported_seq': 11913, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:31.210036+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:31.210036+0000', 'last_peered': '2024-09-11T05:06:31.210036+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:31.210036+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7697568, 'num_objects': 940, 'num_object_clones': 0, 'num_object_copies': 3760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11796, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:33.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:33.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11237", 'reported_seq': 11343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.746217+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:28.746217+0000', 'last_peered': '2024-09-11T05:06:28.746217+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:28.746217+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7559784, 'num_objects': 924, 'num_object_clones': 0, 'num_object_copies': 3696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 924, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11236, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1287}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11541", 'reported_seq': 11650, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:31.120171+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:06:31.120171+0000', 'last_peered': '2024-09-11T05:06:31.120171+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:31.120171+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7165088, 'num_objects': 875, 'num_object_clones': 0, 'num_object_copies': 3500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11539, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11576", 'reported_seq': 11700, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.923741+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:28.923741+0000', 'last_peered': '2024-09-11T05:06:28.923741+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:28.923741+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7820448, 'num_objects': 955, 'num_object_clones': 0, 'num_object_copies': 3820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11574, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1266}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11663", 'reported_seq': 11769, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.705940+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:28.705940+0000', 'last_peered': '2024-09-11T05:06:28.705940+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:28.705940+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8003584, 'num_objects': 977, 'num_object_clones': 0, 'num_object_copies': 3908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11663, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1317}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11767", 'reported_seq': 11862, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:31.261337+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:31.261337+0000', 'last_peered': '2024-09-11T05:06:31.261337+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:31.261337+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7885984, 'num_objects': 963, 'num_object_clones': 0, 'num_object_copies': 3852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11765, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1344}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11781", 'reported_seq': 11905, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.784422+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:28.784422+0000', 'last_peered': '2024-09-11T05:06:28.784422+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:28.784422+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8243520, 'num_objects': 1007, 'num_object_clones': 0, 'num_object_copies': 4028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11778, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1359}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:33.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11592", 'reported_seq': 11705, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.914204+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:28.914204+0000', 'last_peered': '2024-09-11T05:06:28.914204+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:28.914204+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7664800, 'num_objects': 936, 'num_object_clones': 0, 'num_object_copies': 3744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11590, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1370}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11681", 'reported_seq': 11799, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.889320+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:28.889320+0000', 'last_peered': '2024-09-11T05:06:28.889320+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:28.889320+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7958256, 'num_objects': 972, 'num_object_clones': 0, 'num_object_copies': 3888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1326}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11721", 'reported_seq': 16290, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.989785+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:06:28.989785+0000', 'last_peered': '2024-09-11T05:06:28.989785+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:06:28.989785+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8206928, 'num_objects': 1002, 'num_object_clones': 0, 'num_object_copies': 4008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 92, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11720, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2242, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11599", 'reported_seq': 11705, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.870110+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:28.870110+0000', 'last_peered': '2024-09-11T05:06:28.870110+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:28.870110+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8131744, 'num_objects': 993, 'num_object_clones': 0, 'num_object_copies': 3972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11597, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1271}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11495", 'reported_seq': 11615, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:28.706788+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:28.706788+0000', 'last_peered': '2024-09-11T05:06:28.706788+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:28.706788+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7484576, 'num_objects': 914, 'num_object_clones': 0, 'num_object_copies': 3656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11493, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:33.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11846", 'reported_seq': 11966, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:31.259859+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:31.259859+0000', 'last_peered': '2024-09-11T05:06:31.259859+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:31.259859+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7623840, 'num_objects': 931, 'num_object_clones': 0, 'num_object_copies': 3724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11844, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1357}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:33.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:33.850 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29783/61348 objects degraded (48.548%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":720,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":15337,"data_bytes":125596136,"bytes_used":2833588224,"bytes_avail":383713468416,"bytes_total":386547056640,"degraded_objects":29783,"degraded_total":61348,"degraded_ratio":0.48547629914585644,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:33.891 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:06:33.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:34.522 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:34.555 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-11T05:06:33.896865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125596136,"num_objects":15337,"num_object_clones":0,"num_object_copies":61348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170566,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2525,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160970,"ondisk_log_size":160970,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767176,"kb_used_data":247588,"kb_used_omap":14,"kb_used_meta":2519473,"kb_avail":374720184,"statfs":{"total":386547056640,"available":383713468416,"internally_reserved":0,"allocated":253530112,"data_stored":252773304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2579940918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,4,11,7,17,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15679488,"num_objects":-1914,"num_object_clones":0,"num_object_copies":-7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1914,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005209"},"pg_stats":[{"pgid":"1.f","version":"35'11525","reported_seq":11620,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.864814+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:28.864814+0000","last_peered":"2024-09-11T05:06:28.864814+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:28.864814+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101888,"num_objects":989,"num_object_clones":0,"num_object_copies":3956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11525,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11798","reported_seq":11913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.210036+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:31.210036+0000","last_peered":"2024-09-11T05:06:31.210036+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:31.210036+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7697568,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11796,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11237","reported_seq":11343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.746217+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:28.746217+0000","last_peered":"2024-09-11T05:06:28.746217+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:28.746217+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7559784,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11236,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11541","reported_seq":11650,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.120171+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:31.120171+0000","last_peered":"2024-09-11T05:06:31.120171+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:31.120171+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7165088,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11576","reported_seq":11700,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.923741+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:28.923741+0000","last_peered":"2024-09-11T05:06:28.923741+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:28.923741+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820448,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11574,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11663","reported_seq":11769,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.705940+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:28.705940+0000","last_peered":"2024-09-11T05:06:28.705940+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:28.705940+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8003584,"num_objects":977,"num_object_clones":0,"num_object_copies":3908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11663,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11767","reported_seq":11862,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.261337+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:31.261337+0000","last_peered":"2024-09-11T05:06:31.261337+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:31.261337+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7885984,"num_objects":963,"num_object_clones":0,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11765,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1344}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11843","reported_seq":16701,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:29.678930+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:29.678930+0000","last_peered":"2024-09-11T05:06:29.678930+0000","last_clean":"2024-09-11T05:06:29.678930+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:29.678930+0000","last_undegraded":"2024-09-11T05:06:29.678930+0000","last_fullsized":"2024-09-11T05:06:29.678930+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8028160,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11781","reported_seq":11905,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.784422+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:28.784422+0000","last_peered":"2024-09-11T05:06:28.784422+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:28.784422+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8243520,"num_objects":1007,"num_object_clones":0,"num_object_copies":4028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11778,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11592","reported_seq":11705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.914204+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:28.914204+0000","last_peered":"2024-09-11T05:06:28.914204+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:28.914204+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7664800,"num_objects":936,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11681","reported_seq":11799,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.889320+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:28.889320+0000","last_peered":"2024-09-11T05:06:28.889320+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:28.889320+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7958256,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11405","reported_seq":16007,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:29.678949+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:29.678949+0000","last_peered":"2024-09-11T05:06:29.678949+0000","last_clean":"2024-09-11T05:06:29.678949+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:29.678949+0000","last_undegraded":"2024-09-11T05:06:29.678949+0000","last_fullsized":"2024-09-11T05:06:29.678949+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8019968,"num_objects":979,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11721","reported_seq":16290,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.989785+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:28.989785+0000","last_peered":"2024-09-11T05:06:28.989785+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:28.989785+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8206928,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11599","reported_seq":11705,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.870110+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:28.870110+0000","last_peered":"2024-09-11T05:06:28.870110+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:28.870110+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8131744,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11597,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1271}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11495","reported_seq":11615,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:28.706788+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:28.706788+0000","last_peered":"2024-09-11T05:06:28.706788+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:28.706788+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7484576,"num_objects":914,"num_object_clones":0,"num_object_copies":3656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11846","reported_seq":11966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:31.259859+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:31.259859+0000","last_peered":"2024-09-11T05:06:31.259859+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:31.259859+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7623840,"num_objects":931,"num_object_clones":0,"num_object_copies":3724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":125596136,"num_objects":15337,"num_object_clones":0,"num_object_copies":61348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170566,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2525,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250724352,"data_stored":250724352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160970,"ondisk_log_size":160970,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738164,"kb_used_data":62308,"kb_used_omap":4,"kb_used_meta":675835,"kb_avail":93633676,"statfs":{"total":96636764160,"available":95880884224,"internally_reserved":0,"allocated":63803392,"data_stored":63614190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":692055760},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,7,0,4,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715320,"kb_used_data":62180,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93656520,"statfs":{"total":96636764160,"available":95904276480,"internally_reserved":0,"allocated":63672320,"data_stored":63483118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599264,"kb_used_data":62276,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93772576,"statfs":{"total":96636764160,"available":96023117824,"internally_reserved":0,"allocated":63770624,"data_stored":63581422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549842642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714428,"kb_used_data":60824,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93657412,"statfs":{"total":96636764160,"available":95905189888,"internally_reserved":0,"allocated":62283776,"data_stored":62094574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61370368,"data_stored":61370368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62754816,"data_stored":62754816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63275008,"data_stored":63275008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63324160,"data_stored":63324160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:37.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:38.154 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:38.180 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-11T05:06:38.082535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117780968,"num_objects":14383,"num_object_clones":0,"num_object_copies":57532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171520,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2582,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785052,"kb_used_data":232132,"kb_used_omap":14,"kb_used_meta":2552817,"kb_avail":374702308,"statfs":{"total":386547056640,"available":383695163392,"internally_reserved":0,"allocated":237703168,"data_stored":236946360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2614085174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,7,24,12,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9781248,"num_objects":-1194,"num_object_clones":0,"num_object_copies":-4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-57,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1194,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.187427"},"pg_stats":[{"pgid":"1.f","version":"35'11590","reported_seq":11685,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.869056+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:33.869056+0000","last_peered":"2024-09-11T05:06:33.869056+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:33.869056+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7569408,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11865","reported_seq":11980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.122272+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:36.122272+0000","last_peered":"2024-09-11T05:06:36.122272+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:36.122272+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7148704,"num_objects":873,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11302","reported_seq":11407,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.699906+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:33.699906+0000","last_peered":"2024-09-11T05:06:33.699906+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:33.699906+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7027304,"num_objects":859,"num_object_clones":0,"num_object_copies":3436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11301,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11580","reported_seq":11689,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.180249+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:36.180249+0000","last_peered":"2024-09-11T05:06:36.180249+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:36.180249+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6845600,"num_objects":836,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11578,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1425}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11642","reported_seq":11766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.543257+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:33.543257+0000","last_peered":"2024-09-11T05:06:33.543257+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:33.543257+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7279776,"num_objects":889,"num_object_clones":0,"num_object_copies":3556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11640,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11735","reported_seq":11841,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.665912+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:33.665912+0000","last_peered":"2024-09-11T05:06:33.665912+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:33.665912+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7413760,"num_objects":905,"num_object_clones":0,"num_object_copies":3620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11735,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11826","reported_seq":11921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.161468+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:36.161468+0000","last_peered":"2024-09-11T05:06:36.161468+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:36.161468+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7402656,"num_objects":904,"num_object_clones":0,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11899","reported_seq":16757,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:34.617573+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:34.617573+0000","last_peered":"2024-09-11T05:06:34.617573+0000","last_clean":"2024-09-11T05:06:34.617573+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:34.617573+0000","last_undegraded":"2024-09-11T05:06:34.617573+0000","last_fullsized":"2024-09-11T05:06:34.617573+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7569408,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11851","reported_seq":11975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.957076+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:33.957076+0000","last_peered":"2024-09-11T05:06:33.957076+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:33.957076+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7670080,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11848,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1429}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11637","reported_seq":11750,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:34.032943+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:34.032943+0000","last_peered":"2024-09-11T05:06:34.032943+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:34.032943+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7296160,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1415}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11741","reported_seq":11858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.960787+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:33.960787+0000","last_peered":"2024-09-11T05:06:33.960787+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:33.960787+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7466736,"num_objects":912,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11738,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11459","reported_seq":16061,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:34.828127+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:34.828127+0000","last_peered":"2024-09-11T05:06:34.828127+0000","last_clean":"2024-09-11T05:06:34.828127+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:34.828127+0000","last_undegraded":"2024-09-11T05:06:34.828127+0000","last_fullsized":"2024-09-11T05:06:34.828127+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7577600,"num_objects":925,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11789","reported_seq":16472,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:34.055964+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:34.055964+0000","last_peered":"2024-09-11T05:06:34.055964+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:34.055964+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7649872,"num_objects":934,"num_object_clones":0,"num_object_copies":3736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":899}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11662","reported_seq":11768,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:34.024936+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:34.024936+0000","last_peered":"2024-09-11T05:06:34.024936+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:34.024936+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7615648,"num_objects":930,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11660,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11552","reported_seq":11672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.889862+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:33.889862+0000","last_peered":"2024-09-11T05:06:33.889862+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:33.889862+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7017632,"num_objects":857,"num_object_clones":0,"num_object_copies":3428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11550,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1446}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11894","reported_seq":12015,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.218779+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:36.218779+0000","last_peered":"2024-09-11T05:06:36.218779+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:36.218779+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7230624,"num_objects":883,"num_object_clones":0,"num_object_copies":3532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11892,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117780968,"num_objects":14383,"num_object_clones":0,"num_object_copies":57532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171520,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2582,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":234688512,"data_stored":234688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739896,"kb_used_data":58472,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93631944,"statfs":{"total":96636764160,"available":95879110656,"internally_reserved":0,"allocated":59875328,"data_stored":59686126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,7,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716968,"kb_used_data":58264,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93654872,"statfs":{"total":96636764160,"available":95902588928,"internally_reserved":0,"allocated":59662336,"data_stored":59473134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,7,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599460,"kb_used_data":58324,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93772380,"statfs":{"total":96636764160,"available":96022917120,"internally_reserved":0,"allocated":59723776,"data_stored":59534574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554102482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,3,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728728,"kb_used_data":57072,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93643112,"statfs":{"total":96636764160,"available":95890546688,"internally_reserved":0,"allocated":58441728,"data_stored":58252526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,5,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57442304,"data_stored":57442304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58716160,"data_stored":58716160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":59260928,"data_stored":59260928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":59269120,"data_stored":59269120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:38.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:38.848 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:38.850 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:38.877 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-11T05:06:38.082535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117780968,"num_objects":14383,"num_object_clones":0,"num_object_copies":57532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171520,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2582,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785052,"kb_used_data":232132,"kb_used_omap":14,"kb_used_meta":2552817,"kb_avail":374702308,"statfs":{"total":386547056640,"available":383695163392,"internally_reserved":0,"allocated":237703168,"data_stored":236946360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2614085174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,7,24,12,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9781248,"num_objects":-1194,"num_object_clones":0,"num_object_copies":-4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-57,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1194,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.187427"},"pg_stats":[{"pgid":"1.f","version":"35'11590","reported_seq":11685,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.869056+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:33.869056+0000","last_peered":"2024-09-11T05:06:33.869056+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:33.869056+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7569408,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11865","reported_seq":11980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.122272+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:36.122272+0000","last_peered":"2024-09-11T05:06:36.122272+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:36.122272+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7148704,"num_objects":873,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11302","reported_seq":11407,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.699906+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:33.699906+0000","last_peered":"2024-09-11T05:06:33.699906+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:33.699906+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7027304,"num_objects":859,"num_object_clones":0,"num_object_copies":3436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11301,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11580","reported_seq":11689,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.180249+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:36.180249+0000","last_peered":"2024-09-11T05:06:36.180249+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:36.180249+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6845600,"num_objects":836,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11578,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1425}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11642","reported_seq":11766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.543257+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:33.543257+0000","last_peered":"2024-09-11T05:06:33.543257+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:33.543257+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7279776,"num_objects":889,"num_object_clones":0,"num_object_copies":3556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11640,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11735","reported_seq":11841,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.665912+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:33.665912+0000","last_peered":"2024-09-11T05:06:33.665912+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:33.665912+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7413760,"num_objects":905,"num_object_clones":0,"num_object_copies":3620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11735,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11826","reported_seq":11921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.161468+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:36.161468+0000","last_peered":"2024-09-11T05:06:36.161468+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:36.161468+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7402656,"num_objects":904,"num_object_clones":0,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11899","reported_seq":16757,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:34.617573+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:34.617573+0000","last_peered":"2024-09-11T05:06:34.617573+0000","last_clean":"2024-09-11T05:06:34.617573+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:34.617573+0000","last_undegraded":"2024-09-11T05:06:34.617573+0000","last_fullsized":"2024-09-11T05:06:34.617573+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7569408,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11851","reported_seq":11975,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.957076+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:33.957076+0000","last_peered":"2024-09-11T05:06:33.957076+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:33.957076+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7670080,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11848,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1429}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11637","reported_seq":11750,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:34.032943+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:34.032943+0000","last_peered":"2024-09-11T05:06:34.032943+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:34.032943+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7296160,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1415}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11741","reported_seq":11858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.960787+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:33.960787+0000","last_peered":"2024-09-11T05:06:33.960787+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:33.960787+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7466736,"num_objects":912,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11738,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11459","reported_seq":16061,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:34.828127+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:34.828127+0000","last_peered":"2024-09-11T05:06:34.828127+0000","last_clean":"2024-09-11T05:06:34.828127+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:34.828127+0000","last_undegraded":"2024-09-11T05:06:34.828127+0000","last_fullsized":"2024-09-11T05:06:34.828127+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7577600,"num_objects":925,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11789","reported_seq":16472,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:34.055964+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:34.055964+0000","last_peered":"2024-09-11T05:06:34.055964+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:34.055964+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7649872,"num_objects":934,"num_object_clones":0,"num_object_copies":3736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":899}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11662","reported_seq":11768,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:34.024936+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:34.024936+0000","last_peered":"2024-09-11T05:06:34.024936+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:34.024936+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7615648,"num_objects":930,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11660,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1334}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11552","reported_seq":11672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:33.889862+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:33.889862+0000","last_peered":"2024-09-11T05:06:33.889862+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:33.889862+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7017632,"num_objects":857,"num_object_clones":0,"num_object_copies":3428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11550,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1446}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11894","reported_seq":12015,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.218779+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:36.218779+0000","last_peered":"2024-09-11T05:06:36.218779+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:36.218779+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7230624,"num_objects":883,"num_object_clones":0,"num_object_copies":3532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11892,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117780968,"num_objects":14383,"num_object_clones":0,"num_object_copies":57532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171520,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2582,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":234688512,"data_stored":234688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739896,"kb_used_data":58472,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93631944,"statfs":{"total":96636764160,"available":95879110656,"internally_reserved":0,"allocated":59875328,"data_stored":59686126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,7,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716968,"kb_used_data":58264,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93654872,"statfs":{"total":96636764160,"available":95902588928,"internally_reserved":0,"allocated":59662336,"data_stored":59473134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,7,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599460,"kb_used_data":58324,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93772380,"statfs":{"total":96636764160,"available":96022917120,"internally_reserved":0,"allocated":59723776,"data_stored":59534574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554102482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,3,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728728,"kb_used_data":57072,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93643112,"statfs":{"total":96636764160,"available":95890546688,"internally_reserved":0,"allocated":58441728,"data_stored":58252526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,5,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57442304,"data_stored":57442304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58716160,"data_stored":58716160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":59260928,"data_stored":59260928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":59269120,"data_stored":59269120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:38.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:38.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11590", 'reported_seq': 11685, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:33.869056+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:33.869056+0000', 'last_peered': '2024-09-11T05:06:33.869056+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:33.869056+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7569408, 'num_objects': 924, 'num_object_clones': 0, 'num_object_copies': 3696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11590, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:38.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11865", 'reported_seq': 11980, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:36.122272+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:36.122272+0000', 'last_peered': '2024-09-11T05:06:36.122272+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:36.122272+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7148704, 'num_objects': 873, 'num_object_clones': 0, 'num_object_copies': 3492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11863, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1385}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11302", 'reported_seq': 11407, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:33.699906+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:33.699906+0000', 'last_peered': '2024-09-11T05:06:33.699906+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:33.699906+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7027304, 'num_objects': 859, 'num_object_clones': 0, 'num_object_copies': 3436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 859, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11301, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1352}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11580", 'reported_seq': 11689, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:36.180249+0000', 'last_change': '2024-09-11T04:56:20.575713+0000', 'last_active': '2024-09-11T05:06:36.180249+0000', 'last_peered': '2024-09-11T05:06:36.180249+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:36.180249+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6845600, 'num_objects': 836, 'num_object_clones': 0, 'num_object_copies': 3344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11578, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1425}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11642", 'reported_seq': 11766, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:33.543257+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:33.543257+0000', 'last_peered': '2024-09-11T05:06:33.543257+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:33.543257+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7279776, 'num_objects': 889, 'num_object_clones': 0, 'num_object_copies': 3556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11640, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1332}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:38.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11735", 'reported_seq': 11841, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:33.665912+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:33.665912+0000', 'last_peered': '2024-09-11T05:06:33.665912+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:33.665912+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7413760, 'num_objects': 905, 'num_object_clones': 0, 'num_object_copies': 3620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11735, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1389}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11826", 'reported_seq': 11921, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:36.161468+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:36.161468+0000', 'last_peered': '2024-09-11T05:06:36.161468+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:36.161468+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7402656, 'num_objects': 904, 'num_object_clones': 0, 'num_object_copies': 3616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11824, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1403}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11851", 'reported_seq': 11975, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:33.957076+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:33.957076+0000', 'last_peered': '2024-09-11T05:06:33.957076+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:33.957076+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7670080, 'num_objects': 937, 'num_object_clones': 0, 'num_object_copies': 3748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11848, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1429}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11637", 'reported_seq': 11750, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:34.032943+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:34.032943+0000', 'last_peered': '2024-09-11T05:06:34.032943+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:34.032943+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7296160, 'num_objects': 891, 'num_object_clones': 0, 'num_object_copies': 3564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11635, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1415}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11741", 'reported_seq': 11858, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:33.960787+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:33.960787+0000', 'last_peered': '2024-09-11T05:06:33.960787+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:33.960787+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7466736, 'num_objects': 912, 'num_object_clones': 0, 'num_object_copies': 3648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11738, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1386}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:38.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:06:38.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11789", 'reported_seq': 16472, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:34.055964+0000', 'last_change': '2024-09-11T05:02:57.056321+0000', 'last_active': '2024-09-11T05:06:34.055964+0000', 'last_peered': '2024-09-11T05:06:34.055964+0000', 'last_clean': '2024-09-11T04:54:59.712354+0000', 'last_became_active': '2024-09-11T04:56:20.559471+0000', 'last_became_peered': '2024-09-11T04:56:20.559471+0000', 'last_unstale': '2024-09-11T05:06:34.055964+0000', 'last_undegraded': '2024-09-11T04:56:20.251122+0000', 'last_fullsized': '2024-09-11T04:56:20.224991+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7649872, 'num_objects': 934, 'num_object_clones': 0, 'num_object_copies': 3736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 35, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11788, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2299, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 899}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:38.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:38.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11662", 'reported_seq': 11768, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:34.024936+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:34.024936+0000', 'last_peered': '2024-09-11T05:06:34.024936+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:34.024936+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7615648, 'num_objects': 930, 'num_object_clones': 0, 'num_object_copies': 3720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11660, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:38.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:38.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11552", 'reported_seq': 11672, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:33.889862+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:33.889862+0000', 'last_peered': '2024-09-11T05:06:33.889862+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:33.889862+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7017632, 'num_objects': 857, 'num_object_clones': 0, 'num_object_copies': 3428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11550, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:38.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:38.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11894", 'reported_seq': 12015, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:36.218779+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:36.218779+0000', 'last_peered': '2024-09-11T05:06:36.218779+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:36.218779+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7230624, 'num_objects': 883, 'num_object_clones': 0, 'num_object_copies': 3532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11892, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1405}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:38.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:39.678 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:39.678 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29726/57532 objects degraded (51.669%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":726,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14383,"data_bytes":117780968,"bytes_used":2851893248,"bytes_avail":383695163392,"bytes_total":386547056640,"degraded_objects":29726,"degraded_total":57532,"degraded_ratio":0.51668636584857119,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":189},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:39.714 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:06:39.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:40.313 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:40.314 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:40.347 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-11T05:06:39.899468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":113324520,"num_objects":13839,"num_object_clones":0,"num_object_copies":55356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172064,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2582,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160768,"ondisk_log_size":160768,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786048,"kb_used_data":224792,"kb_used_omap":14,"kb_used_meta":2561137,"kb_avail":374701312,"statfs":{"total":386547056640,"available":383694143488,"internally_reserved":0,"allocated":230187008,"data_stored":229411586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2622604854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,5,12,10,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12271616,"num_objects":-1498,"num_object_clones":0,"num_object_copies":-5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-57,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1498,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003478"},"pg_stats":[{"pgid":"1.f","version":"37'11652","reported_seq":11751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.107177+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:39.107177+0000","last_peered":"2024-09-11T05:06:39.107177+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:39.107177+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7061504,"num_objects":862,"num_object_clones":0,"num_object_copies":3448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1429}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11865","reported_seq":11980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.122272+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:36.122272+0000","last_peered":"2024-09-11T05:06:36.122272+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:36.122272+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7148704,"num_objects":873,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11358","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.968442+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:38.968442+0000","last_peered":"2024-09-11T05:06:38.968442+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:38.968442+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6568552,"num_objects":803,"num_object_clones":0,"num_object_copies":3212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11357,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11580","reported_seq":11689,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.180249+0000","last_change":"2024-09-11T04:56:20.575713+0000","last_active":"2024-09-11T05:06:36.180249+0000","last_peered":"2024-09-11T05:06:36.180249+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:36.180249+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6845600,"num_objects":836,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11578,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1425}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11703","reported_seq":11831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.730975+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:38.730975+0000","last_peered":"2024-09-11T05:06:38.730975+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:38.730975+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6780064,"num_objects":828,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11801","reported_seq":11909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.048568+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:39.048568+0000","last_peered":"2024-09-11T05:06:39.048568+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:39.048568+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6873088,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11801,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11826","reported_seq":11921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.161468+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:36.161468+0000","last_peered":"2024-09-11T05:06:36.161468+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:36.161468+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7402656,"num_objects":904,"num_object_clones":0,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11899","reported_seq":16757,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:34.617573+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:34.617573+0000","last_peered":"2024-09-11T05:06:34.617573+0000","last_clean":"2024-09-11T05:06:34.617573+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:34.617573+0000","last_undegraded":"2024-09-11T05:06:34.617573+0000","last_fullsized":"2024-09-11T05:06:34.617573+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7569408,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11908","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.133948+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:39.133948+0000","last_peered":"2024-09-11T05:06:39.133948+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:39.133948+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7203136,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11905,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11707","reported_seq":11824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.940675+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:38.940675+0000","last_peered":"2024-09-11T05:06:38.940675+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:38.940675+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722720,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11705,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1485}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11797","reported_seq":11919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.027070+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:39.027070+0000","last_peered":"2024-09-11T05:06:39.027070+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:39.027070+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7007984,"num_objects":856,"num_object_clones":0,"num_object_copies":3424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11459","reported_seq":16061,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-11T05:06:34.828127+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:34.828127+0000","last_peered":"2024-09-11T05:06:34.828127+0000","last_clean":"2024-09-11T05:06:34.828127+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:34.828127+0000","last_undegraded":"2024-09-11T05:06:34.828127+0000","last_fullsized":"2024-09-11T05:06:34.828127+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7577600,"num_objects":925,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'11789","reported_seq":16472,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:34.055964+0000","last_change":"2024-09-11T05:02:57.056321+0000","last_active":"2024-09-11T05:06:34.055964+0000","last_peered":"2024-09-11T05:06:34.055964+0000","last_clean":"2024-09-11T04:54:59.712354+0000","last_became_active":"2024-09-11T04:56:20.559471+0000","last_became_peered":"2024-09-11T04:56:20.559471+0000","last_unstale":"2024-09-11T05:06:34.055964+0000","last_undegraded":"2024-09-11T04:56:20.251122+0000","last_fullsized":"2024-09-11T04:56:20.224991+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7649872,"num_objects":934,"num_object_clones":0,"num_object_copies":3736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":899}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11712","reported_seq":11823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.133468+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:39.133468+0000","last_peered":"2024-09-11T05:06:39.133468+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:39.133468+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206048,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1384}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11618","reported_seq":11742,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.033455+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:39.033455+0000","last_peered":"2024-09-11T05:06:39.033455+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:39.033455+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6476960,"num_objects":791,"num_object_clones":0,"num_object_copies":3164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11894","reported_seq":12015,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:36.218779+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:36.218779+0000","last_peered":"2024-09-11T05:06:36.218779+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:36.218779+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7230624,"num_objects":883,"num_object_clones":0,"num_object_copies":3532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11892,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":113324520,"num_objects":13839,"num_object_clones":0,"num_object_copies":55356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172064,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2582,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":227110912,"data_stored":227110912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160768,"ondisk_log_size":160768,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740404,"kb_used_data":54812,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93631436,"statfs":{"total":96636764160,"available":95878590464,"internally_reserved":0,"allocated":56127488,"data_stored":55928979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717456,"kb_used_data":54584,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93654384,"statfs":{"total":96636764160,"available":95902089216,"internally_reserved":0,"allocated":55894016,"data_stored":55695507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599460,"kb_used_data":58324,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93772380,"statfs":{"total":96636764160,"available":96022917120,"internally_reserved":0,"allocated":59723776,"data_stored":59534574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554102482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,3,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728728,"kb_used_data":57072,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93643112,"statfs":{"total":96636764160,"available":95890546688,"internally_reserved":0,"allocated":58441728,"data_stored":58252526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,5,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57442304,"data_stored":57442304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58716160,"data_stored":58716160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":55480320,"data_stored":55480320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":55472128,"data_stored":55472128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:43.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:44.053 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:44.080 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-11T05:06:43.901992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110088680,"num_objects":13444,"num_object_clones":0,"num_object_copies":53776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172459,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2671,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769784,"kb_used_data":216872,"kb_used_omap":14,"kb_used_meta":2552817,"kb_avail":374717576,"statfs":{"total":386547056640,"available":383710797824,"internally_reserved":0,"allocated":222076928,"data_stored":221282892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2614085174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,9,3,10,15,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9437184,"num_objects":-1152,"num_object_clones":0,"num_object_copies":-4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1152,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004316"},"pg_stats":[{"pgid":"1.f","version":"37'11652","reported_seq":11751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.107177+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:39.107177+0000","last_peered":"2024-09-11T05:06:39.107177+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:39.107177+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7061504,"num_objects":862,"num_object_clones":0,"num_object_copies":3448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1429}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11916","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.139403+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:41.139403+0000","last_peered":"2024-09-11T05:06:41.139403+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:41.139403+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6730912,"num_objects":822,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11358","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.968442+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:38.968442+0000","last_peered":"2024-09-11T05:06:38.968442+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:38.968442+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6568552,"num_objects":803,"num_object_clones":0,"num_object_copies":3212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11357,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11628","reported_seq":11850,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.371346+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:41.371346+0000","last_peered":"2024-09-11T05:06:41.371346+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:41.371346+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6452384,"num_objects":788,"num_object_clones":0,"num_object_copies":3152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11703","reported_seq":11831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.730975+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:38.730975+0000","last_peered":"2024-09-11T05:06:38.730975+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:38.730975+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6780064,"num_objects":828,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11801","reported_seq":11909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.048568+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:39.048568+0000","last_peered":"2024-09-11T05:06:39.048568+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:39.048568+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6873088,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11801,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11881","reported_seq":11980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.320293+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:41.320293+0000","last_peered":"2024-09-11T05:06:41.320293+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:41.320293+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6952096,"num_objects":849,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1458}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11956","reported_seq":16818,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:40.002081+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:40.002081+0000","last_peered":"2024-09-11T05:06:40.002081+0000","last_clean":"2024-09-11T05:06:40.002081+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:40.002081+0000","last_undegraded":"2024-09-11T05:06:40.002081+0000","last_fullsized":"2024-09-11T05:06:40.002081+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7102464,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11908","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.133948+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:39.133948+0000","last_peered":"2024-09-11T05:06:39.133948+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:39.133948+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7203136,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11905,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11707","reported_seq":11824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.940675+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:38.940675+0000","last_peered":"2024-09-11T05:06:38.940675+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:38.940675+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722720,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11705,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1485}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11797","reported_seq":11919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.027070+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:39.027070+0000","last_peered":"2024-09-11T05:06:39.027070+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:39.027070+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7007984,"num_objects":856,"num_object_clones":0,"num_object_copies":3424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11529","reported_seq":16134,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:40.034656+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:40.034656+0000","last_peered":"2024-09-11T05:06:40.034656+0000","last_clean":"2024-09-11T05:06:40.034656+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:40.034656+0000","last_undegraded":"2024-09-11T05:06:40.034656+0000","last_fullsized":"2024-09-11T05:06:40.034656+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7004160,"num_objects":855,"num_object_clones":0,"num_object_copies":3420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11854","reported_seq":16629,"reported_epoch":37,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T05:06:40.094114+0000","last_change":"2024-09-11T05:06:38.988745+0000","last_active":"2024-09-11T05:06:40.094114+0000","last_peered":"2024-09-11T05:06:40.094114+0000","last_clean":"2024-09-11T05:06:40.094114+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:40.094114+0000","last_undegraded":"2024-09-11T05:06:40.094114+0000","last_fullsized":"2024-09-11T05:06:40.094114+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":598,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7117392,"num_objects":869,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11853,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11712","reported_seq":11823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.133468+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:39.133468+0000","last_peered":"2024-09-11T05:06:39.133468+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:39.133468+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206048,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1384}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11618","reported_seq":11742,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.033455+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:39.033455+0000","last_peered":"2024-09-11T05:06:39.033455+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:39.033455+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6476960,"num_objects":791,"num_object_clones":0,"num_object_copies":3164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11943","reported_seq":12068,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.214327+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:41.214327+0000","last_peered":"2024-09-11T05:06:41.214327+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:41.214327+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6829216,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11941,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":110088680,"num_objects":13444,"num_object_clones":0,"num_object_copies":53776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172459,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2671,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":219607040,"data_stored":219607040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740404,"kb_used_data":54812,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93631436,"statfs":{"total":96636764160,"available":95878590464,"internally_reserved":0,"allocated":56127488,"data_stored":55928979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717456,"kb_used_data":54584,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93654384,"statfs":{"total":96636764160,"available":95902089216,"internally_reserved":0,"allocated":55894016,"data_stored":55695507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599316,"kb_used_data":54020,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93772524,"statfs":{"total":96636764160,"available":96023064576,"internally_reserved":0,"allocated":55316480,"data_stored":55117971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558362322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,7,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712608,"kb_used_data":53456,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93659232,"statfs":{"total":96636764160,"available":95907053568,"internally_reserved":0,"allocated":54738944,"data_stored":54540435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53813248,"data_stored":53813248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54841344,"data_stored":54841344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":55480320,"data_stored":55480320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":55472128,"data_stored":55472128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:44.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:44.728 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:44.745 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-11T05:06:43.901992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110088680,"num_objects":13444,"num_object_clones":0,"num_object_copies":53776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172459,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2671,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769784,"kb_used_data":216872,"kb_used_omap":14,"kb_used_meta":2552817,"kb_avail":374717576,"statfs":{"total":386547056640,"available":383710797824,"internally_reserved":0,"allocated":222076928,"data_stored":221282892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2614085174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,9,3,10,15,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9437184,"num_objects":-1152,"num_object_clones":0,"num_object_copies":-4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1152,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004316"},"pg_stats":[{"pgid":"1.f","version":"37'11652","reported_seq":11751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.107177+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:39.107177+0000","last_peered":"2024-09-11T05:06:39.107177+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:39.107177+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7061504,"num_objects":862,"num_object_clones":0,"num_object_copies":3448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1429}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11916","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.139403+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:41.139403+0000","last_peered":"2024-09-11T05:06:41.139403+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:41.139403+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6730912,"num_objects":822,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11358","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.968442+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:38.968442+0000","last_peered":"2024-09-11T05:06:38.968442+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:38.968442+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6568552,"num_objects":803,"num_object_clones":0,"num_object_copies":3212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11357,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11628","reported_seq":11850,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.371346+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:41.371346+0000","last_peered":"2024-09-11T05:06:41.371346+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:41.371346+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6452384,"num_objects":788,"num_object_clones":0,"num_object_copies":3152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11703","reported_seq":11831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.730975+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:38.730975+0000","last_peered":"2024-09-11T05:06:38.730975+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:38.730975+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6780064,"num_objects":828,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11801","reported_seq":11909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.048568+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:39.048568+0000","last_peered":"2024-09-11T05:06:39.048568+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:39.048568+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6873088,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11801,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11881","reported_seq":11980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.320293+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:41.320293+0000","last_peered":"2024-09-11T05:06:41.320293+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:41.320293+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6952096,"num_objects":849,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1458}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11956","reported_seq":16818,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:40.002081+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:40.002081+0000","last_peered":"2024-09-11T05:06:40.002081+0000","last_clean":"2024-09-11T05:06:40.002081+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:40.002081+0000","last_undegraded":"2024-09-11T05:06:40.002081+0000","last_fullsized":"2024-09-11T05:06:40.002081+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7102464,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11908","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.133948+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:39.133948+0000","last_peered":"2024-09-11T05:06:39.133948+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:39.133948+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7203136,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11905,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11707","reported_seq":11824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:38.940675+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:38.940675+0000","last_peered":"2024-09-11T05:06:38.940675+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:38.940675+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722720,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11705,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1485}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11797","reported_seq":11919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.027070+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:39.027070+0000","last_peered":"2024-09-11T05:06:39.027070+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:39.027070+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7007984,"num_objects":856,"num_object_clones":0,"num_object_copies":3424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11529","reported_seq":16134,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:40.034656+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:40.034656+0000","last_peered":"2024-09-11T05:06:40.034656+0000","last_clean":"2024-09-11T05:06:40.034656+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:40.034656+0000","last_undegraded":"2024-09-11T05:06:40.034656+0000","last_fullsized":"2024-09-11T05:06:40.034656+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7004160,"num_objects":855,"num_object_clones":0,"num_object_copies":3420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11854","reported_seq":16629,"reported_epoch":37,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T05:06:40.094114+0000","last_change":"2024-09-11T05:06:38.988745+0000","last_active":"2024-09-11T05:06:40.094114+0000","last_peered":"2024-09-11T05:06:40.094114+0000","last_clean":"2024-09-11T05:06:40.094114+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:40.094114+0000","last_undegraded":"2024-09-11T05:06:40.094114+0000","last_fullsized":"2024-09-11T05:06:40.094114+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":598,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7117392,"num_objects":869,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11853,"num_write_kb":50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11712","reported_seq":11823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.133468+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:39.133468+0000","last_peered":"2024-09-11T05:06:39.133468+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:39.133468+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206048,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1384}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11618","reported_seq":11742,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:39.033455+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:39.033455+0000","last_peered":"2024-09-11T05:06:39.033455+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:39.033455+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6476960,"num_objects":791,"num_object_clones":0,"num_object_copies":3164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11943","reported_seq":12068,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.214327+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:41.214327+0000","last_peered":"2024-09-11T05:06:41.214327+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:41.214327+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6829216,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11941,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":110088680,"num_objects":13444,"num_object_clones":0,"num_object_copies":53776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172459,"num_write_kb":704712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2671,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":219607040,"data_stored":219607040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740404,"kb_used_data":54812,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93631436,"statfs":{"total":96636764160,"available":95878590464,"internally_reserved":0,"allocated":56127488,"data_stored":55928979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717456,"kb_used_data":54584,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93654384,"statfs":{"total":96636764160,"available":95902089216,"internally_reserved":0,"allocated":55894016,"data_stored":55695507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599316,"kb_used_data":54020,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93772524,"statfs":{"total":96636764160,"available":96023064576,"internally_reserved":0,"allocated":55316480,"data_stored":55117971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558362322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,7,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712608,"kb_used_data":53456,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93659232,"statfs":{"total":96636764160,"available":95907053568,"internally_reserved":0,"allocated":54738944,"data_stored":54540435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53813248,"data_stored":53813248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54841344,"data_stored":54841344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":55480320,"data_stored":55480320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":55472128,"data_stored":55472128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11652", 'reported_seq': 11751, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:39.107177+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:39.107177+0000', 'last_peered': '2024-09-11T05:06:39.107177+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:39.107177+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7061504, 'num_objects': 862, 'num_object_clones': 0, 'num_object_copies': 3448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11652, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1429}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11916", 'reported_seq': 12035, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:41.139403+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:41.139403+0000', 'last_peered': '2024-09-11T05:06:41.139403+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:41.139403+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6730912, 'num_objects': 822, 'num_object_clones': 0, 'num_object_copies': 3288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11914, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11358", 'reported_seq': 11468, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:38.968442+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:38.968442+0000', 'last_peered': '2024-09-11T05:06:38.968442+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:38.968442+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6568552, 'num_objects': 803, 'num_object_clones': 0, 'num_object_copies': 3212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 803, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11357, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1408}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11628", 'reported_seq': 11850, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:41.371346+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:06:41.371346+0000', 'last_peered': '2024-09-11T05:06:41.371346+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:41.371346+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6452384, 'num_objects': 788, 'num_object_clones': 0, 'num_object_copies': 3152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2209, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11626, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:44.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11703", 'reported_seq': 11831, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:38.730975+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:38.730975+0000', 'last_peered': '2024-09-11T05:06:38.730975+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:38.730975+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6780064, 'num_objects': 828, 'num_object_clones': 0, 'num_object_copies': 3312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 828, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11701, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1393}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11801", 'reported_seq': 11909, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:39.048568+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:39.048568+0000', 'last_peered': '2024-09-11T05:06:39.048568+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:39.048568+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6873088, 'num_objects': 839, 'num_object_clones': 0, 'num_object_copies': 3356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11801, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1455}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11881", 'reported_seq': 11980, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:41.320293+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:41.320293+0000', 'last_peered': '2024-09-11T05:06:41.320293+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:41.320293+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6952096, 'num_objects': 849, 'num_object_clones': 0, 'num_object_copies': 3396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1458}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11908", 'reported_seq': 12035, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:39.133948+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:39.133948+0000', 'last_peered': '2024-09-11T05:06:39.133948+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:39.133948+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7203136, 'num_objects': 880, 'num_object_clones': 0, 'num_object_copies': 3520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11905, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1486}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11707", 'reported_seq': 11824, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:38.940675+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:38.940675+0000', 'last_peered': '2024-09-11T05:06:38.940675+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:38.940675+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6722720, 'num_objects': 821, 'num_object_clones': 0, 'num_object_copies': 3284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11705, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1485}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:44.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11797", 'reported_seq': 11919, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:39.027070+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:39.027070+0000', 'last_peered': '2024-09-11T05:06:39.027070+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:39.027070+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7007984, 'num_objects': 856, 'num_object_clones': 0, 'num_object_copies': 3424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11794, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1442}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:44.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-11T05:06:44.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11854", 'reported_seq': 16629, 'reported_epoch': 37, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-11T05:06:40.094114+0000', 'last_change': '2024-09-11T05:06:38.988745+0000', 'last_active': '2024-09-11T05:06:40.094114+0000', 'last_peered': '2024-09-11T05:06:40.094114+0000', 'last_clean': '2024-09-11T05:06:40.094114+0000', 'last_became_active': '2024-09-11T05:06:38.153760+0000', 'last_became_peered': '2024-09-11T05:06:38.153760+0000', 'last_unstale': '2024-09-11T05:06:40.094114+0000', 'last_undegraded': '2024-09-11T05:06:40.094114+0000', 'last_fullsized': '2024-09-11T05:06:40.094114+0000', 'mapping_epoch': 36, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 37, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 598, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 1s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7117392, 'num_objects': 869, 'num_object_clones': 0, 'num_object_copies': 3476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11853, 'num_write_kb': 50887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2334, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [1, 2, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-11T05:06:44.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:44.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11712", 'reported_seq': 11823, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:39.133468+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:39.133468+0000', 'last_peered': '2024-09-11T05:06:39.133468+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:39.133468+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7206048, 'num_objects': 880, 'num_object_clones': 0, 'num_object_copies': 3520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11710, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:44.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:44.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11618", 'reported_seq': 11742, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:39.033455+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:39.033455+0000', 'last_peered': '2024-09-11T05:06:39.033455+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:39.033455+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6476960, 'num_objects': 791, 'num_object_clones': 0, 'num_object_copies': 3164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11616, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:44.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:44.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11943", 'reported_seq': 12068, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:41.214327+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:41.214327+0000', 'last_peered': '2024-09-11T05:06:41.214327+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:41.214327+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6829216, 'num_objects': 834, 'num_object_clones': 0, 'num_object_copies': 3336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11941, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1454}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:44.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:45.505 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:45.505 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29637/53776 objects degraded (55.112%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":732,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13444,"data_bytes":110088680,"bytes_used":2836258816,"bytes_avail":383710797824,"bytes_total":386547056640,"degraded_objects":29637,"degraded_total":53776,"degraded_ratio":0.55111945849449573,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":143},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:45.530 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:06:45.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:46.184 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:46.208 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-11T05:06:45.902789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103856024,"num_objects":12683,"num_object_clones":0,"num_object_copies":50732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161367,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767752,"kb_used_data":205116,"kb_used_omap":14,"kb_used_meta":2562545,"kb_avail":374719608,"statfs":{"total":386547056640,"available":383712878592,"internally_reserved":0,"allocated":210038784,"data_stored":209244748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2624046646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,8,8,16,11,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13924944,"num_objects":-1700,"num_object_clones":0,"num_object_copies":-6800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-10153,"num_write_kb":-50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.850565"},"pg_stats":[{"pgid":"1.f","version":"37'11720","reported_seq":11819,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.116034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:44.116034+0000","last_peered":"2024-09-11T05:06:44.116034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:44.116034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6504448,"num_objects":794,"num_object_clones":0,"num_object_copies":3176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11916","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.139403+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:41.139403+0000","last_peered":"2024-09-11T05:06:41.139403+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:41.139403+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6730912,"num_objects":822,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11411","reported_seq":11521,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.027996+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:44.027996+0000","last_peered":"2024-09-11T05:06:44.027996+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:44.027996+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134376,"num_objects":750,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11410,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11628","reported_seq":11850,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.371346+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:41.371346+0000","last_peered":"2024-09-11T05:06:41.371346+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:41.371346+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6452384,"num_objects":788,"num_object_clones":0,"num_object_copies":3152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11766","reported_seq":11894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.126235+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:44.126235+0000","last_peered":"2024-09-11T05:06:44.126235+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:44.126235+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6263968,"num_objects":765,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11854","reported_seq":11964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.116640+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:44.116640+0000","last_peered":"2024-09-11T05:06:44.116640+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:44.116640+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6438912,"num_objects":786,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11854,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11881","reported_seq":11980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.320293+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:41.320293+0000","last_peered":"2024-09-11T05:06:41.320293+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:41.320293+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6952096,"num_objects":849,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1458}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12006","reported_seq":16868,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.220976+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:45.220976+0000","last_peered":"2024-09-11T05:06:45.220976+0000","last_clean":"2024-09-11T05:06:45.220976+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:45.220976+0000","last_undegraded":"2024-09-11T05:06:45.220976+0000","last_fullsized":"2024-09-11T05:06:45.220976+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11978","reported_seq":12105,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.052041+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:44.052041+0000","last_peered":"2024-09-11T05:06:44.052041+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:44.052041+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6629696,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1556}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11778","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.176141+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:44.176141+0000","last_peered":"2024-09-11T05:06:44.176141+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:44.176141+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6141088,"num_objects":750,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11856","reported_seq":11977,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.139087+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:44.139087+0000","last_peered":"2024-09-11T05:06:44.139087+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:44.139087+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6524656,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11853,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11602","reported_seq":16208,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.340724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:45.340724+0000","last_peered":"2024-09-11T05:06:45.340724+0000","last_clean":"2024-09-11T05:06:45.340724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:45.340724+0000","last_undegraded":"2024-09-11T05:06:45.340724+0000","last_fullsized":"2024-09-11T05:06:45.340724+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406144,"num_objects":782,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5412,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11915","reported_seq":16696,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.379926+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:06:45.379926+0000","last_peered":"2024-09-11T05:06:45.379926+0000","last_clean":"2024-09-11T05:06:45.379926+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:45.379926+0000","last_undegraded":"2024-09-11T05:06:45.379926+0000","last_fullsized":"2024-09-11T05:06:45.379926+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11790","reported_seq":11902,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.056805+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:44.056805+0000","last_peered":"2024-09-11T05:06:44.056805+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:44.056805+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6567072,"num_objects":802,"num_object_clones":0,"num_object_copies":3208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1462}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11679","reported_seq":11803,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:43.593838+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:43.593838+0000","last_peered":"2024-09-11T05:06:43.593838+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:43.593838+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5977248,"num_objects":730,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11677,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1573}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11943","reported_seq":12068,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:41.214327+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:41.214327+0000","last_peered":"2024-09-11T05:06:41.214327+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:41.214327+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6829216,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11941,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":103856024,"num_objects":12683,"num_object_clones":0,"num_object_copies":50732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161367,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207499264,"data_stored":207499264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736612,"kb_used_data":51020,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93635228,"statfs":{"total":96636764160,"available":95882473472,"internally_reserved":0,"allocated":52244480,"data_stored":52045971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,9,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735712,"kb_used_data":50636,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93636128,"statfs":{"total":96636764160,"available":95883395072,"internally_reserved":0,"allocated":51851264,"data_stored":51652755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582820,"kb_used_data":50004,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93789020,"statfs":{"total":96636764160,"available":96039956480,"internally_reserved":0,"allocated":51204096,"data_stored":51005587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545582802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,4,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712608,"kb_used_data":53456,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93659232,"statfs":{"total":96636764160,"available":95907053568,"internally_reserved":0,"allocated":54738944,"data_stored":54540435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53813248,"data_stored":53813248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50733056,"data_stored":50733056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51486720,"data_stored":51486720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51466240,"data_stored":51466240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:49.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:49.874 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:49.875 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:49.898 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-11T05:06:47.903747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101889944,"num_objects":12443,"num_object_clones":0,"num_object_copies":49772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161607,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786132,"kb_used_data":201288,"kb_used_omap":14,"kb_used_meta":2584753,"kb_avail":374701228,"statfs":{"total":386547056640,"available":383694057472,"internally_reserved":0,"allocated":206118912,"data_stored":205324876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2646787638},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,3,4,8,16,11,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15891024,"num_objects":-1940,"num_object_clones":0,"num_object_copies":-7760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9913,"num_write_kb":-50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005201"},"pg_stats":[{"pgid":"1.f","version":"37'11720","reported_seq":11819,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.116034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:44.116034+0000","last_peered":"2024-09-11T05:06:44.116034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:44.116034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6504448,"num_objects":794,"num_object_clones":0,"num_object_copies":3176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11983","reported_seq":12101,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:46.515338+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:46.515338+0000","last_peered":"2024-09-11T05:06:46.515338+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:46.515338+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6182048,"num_objects":755,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11411","reported_seq":11521,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.027996+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:44.027996+0000","last_peered":"2024-09-11T05:06:44.027996+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:44.027996+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134376,"num_objects":750,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11410,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11680","reported_seq":11999,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:46.488812+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:46.488812+0000","last_peered":"2024-09-11T05:06:46.488812+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:46.488812+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6026400,"num_objects":736,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11766","reported_seq":11894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.126235+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:44.126235+0000","last_peered":"2024-09-11T05:06:44.126235+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:44.126235+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6263968,"num_objects":765,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11854","reported_seq":11964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.116640+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:44.116640+0000","last_peered":"2024-09-11T05:06:44.116640+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:44.116640+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6438912,"num_objects":786,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11854,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11945","reported_seq":12044,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:46.369637+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:46.369637+0000","last_peered":"2024-09-11T05:06:46.369637+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:46.369637+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6427808,"num_objects":785,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11943,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1522}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12006","reported_seq":16868,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.220976+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:45.220976+0000","last_peered":"2024-09-11T05:06:45.220976+0000","last_clean":"2024-09-11T05:06:45.220976+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:45.220976+0000","last_undegraded":"2024-09-11T05:06:45.220976+0000","last_fullsized":"2024-09-11T05:06:45.220976+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11978","reported_seq":12105,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.052041+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:44.052041+0000","last_peered":"2024-09-11T05:06:44.052041+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:44.052041+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6629696,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1556}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11778","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.176141+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:44.176141+0000","last_peered":"2024-09-11T05:06:44.176141+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:44.176141+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6141088,"num_objects":750,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11856","reported_seq":11977,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.139087+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:44.139087+0000","last_peered":"2024-09-11T05:06:44.139087+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:44.139087+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6524656,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11853,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11602","reported_seq":16208,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.340724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:45.340724+0000","last_peered":"2024-09-11T05:06:45.340724+0000","last_clean":"2024-09-11T05:06:45.340724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:45.340724+0000","last_undegraded":"2024-09-11T05:06:45.340724+0000","last_fullsized":"2024-09-11T05:06:45.340724+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406144,"num_objects":782,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5412,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11915","reported_seq":16696,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.379926+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:06:45.379926+0000","last_peered":"2024-09-11T05:06:45.379926+0000","last_clean":"2024-09-11T05:06:45.379926+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:45.379926+0000","last_undegraded":"2024-09-11T05:06:45.379926+0000","last_fullsized":"2024-09-11T05:06:45.379926+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11790","reported_seq":11902,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:44.056805+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:44.056805+0000","last_peered":"2024-09-11T05:06:44.056805+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:44.056805+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6567072,"num_objects":802,"num_object_clones":0,"num_object_copies":3208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1462}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11679","reported_seq":11803,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:43.593838+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:43.593838+0000","last_peered":"2024-09-11T05:06:43.593838+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:43.593838+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5977248,"num_objects":730,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11677,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1573}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12000","reported_seq":12125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:46.156356+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:46.156356+0000","last_peered":"2024-09-11T05:06:46.156356+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:46.156356+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6362272,"num_objects":777,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101889944,"num_objects":12443,"num_object_clones":0,"num_object_copies":49772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161607,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203493376,"data_stored":203493376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736612,"kb_used_data":51020,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93635228,"statfs":{"total":96636764160,"available":95882473472,"internally_reserved":0,"allocated":52244480,"data_stored":52045971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,9,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735712,"kb_used_data":50636,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93636128,"statfs":{"total":96636764160,"available":95883395072,"internally_reserved":0,"allocated":51851264,"data_stored":51652755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582820,"kb_used_data":50004,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93789020,"statfs":{"total":96636764160,"available":96039956480,"internally_reserved":0,"allocated":51204096,"data_stored":51005587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545582802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,4,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730988,"kb_used_data":49628,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93640852,"statfs":{"total":96636764160,"available":95888232448,"internally_reserved":0,"allocated":50819072,"data_stored":50620563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49807360,"data_stored":49807360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50733056,"data_stored":50733056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51486720,"data_stored":51486720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51466240,"data_stored":51466240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:49.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:50.555 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:50.556 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:50.577 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-11T05:06:49.904530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97507224,"num_objects":11908,"num_object_clones":0,"num_object_copies":47632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162142,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160698,"ondisk_log_size":160698,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804848,"kb_used_data":193636,"kb_used_omap":14,"kb_used_meta":2611121,"kb_avail":374682512,"statfs":{"total":386547056640,"available":383674892288,"internally_reserved":0,"allocated":198283264,"data_stored":197489228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2673788470},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,9,12,14,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20273744,"num_objects":-2475,"num_object_clones":0,"num_object_copies":-9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9378,"num_write_kb":-50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.821994"},"pg_stats":[{"pgid":"1.f","version":"37'11781","reported_seq":11880,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.159403+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:49.159403+0000","last_peered":"2024-09-11T05:06:49.159403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:49.159403+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11983","reported_seq":12101,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:46.515338+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:46.515338+0000","last_peered":"2024-09-11T05:06:46.515338+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:46.515338+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6182048,"num_objects":755,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11467","reported_seq":11577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.035162+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:49.035162+0000","last_peered":"2024-09-11T05:06:49.035162+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:49.035162+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5675624,"num_objects":694,"num_object_clones":0,"num_object_copies":2776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11466,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11680","reported_seq":11999,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:46.488812+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:46.488812+0000","last_peered":"2024-09-11T05:06:46.488812+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:46.488812+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6026400,"num_objects":736,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11826","reported_seq":11954,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.146267+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:49.146267+0000","last_peered":"2024-09-11T05:06:49.146267+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:49.146267+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5772448,"num_objects":705,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11917","reported_seq":12027,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.302053+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:49.302053+0000","last_peered":"2024-09-11T05:06:49.302053+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:49.302053+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":723,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11945","reported_seq":12044,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:46.369637+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:46.369637+0000","last_peered":"2024-09-11T05:06:46.369637+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:46.369637+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6427808,"num_objects":785,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11943,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1522}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12006","reported_seq":16868,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.220976+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:45.220976+0000","last_peered":"2024-09-11T05:06:45.220976+0000","last_clean":"2024-09-11T05:06:45.220976+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:45.220976+0000","last_undegraded":"2024-09-11T05:06:45.220976+0000","last_fullsized":"2024-09-11T05:06:45.220976+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6692864,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12029","reported_seq":12157,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.057935+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:49.057935+0000","last_peered":"2024-09-11T05:06:49.057935+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:49.057935+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6211904,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12026,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11835","reported_seq":11952,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.033379+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:49.033379+0000","last_peered":"2024-09-11T05:06:49.033379+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:49.033379+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5674144,"num_objects":693,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11917","reported_seq":12039,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.239646+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:49.239646+0000","last_peered":"2024-09-11T05:06:49.239646+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:49.239646+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6024944,"num_objects":736,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11602","reported_seq":16208,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.340724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:45.340724+0000","last_peered":"2024-09-11T05:06:45.340724+0000","last_clean":"2024-09-11T05:06:45.340724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:45.340724+0000","last_undegraded":"2024-09-11T05:06:45.340724+0000","last_fullsized":"2024-09-11T05:06:45.340724+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406144,"num_objects":782,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5412,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11915","reported_seq":16696,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:45.379926+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:06:45.379926+0000","last_peered":"2024-09-11T05:06:45.379926+0000","last_clean":"2024-09-11T05:06:45.379926+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:45.379926+0000","last_undegraded":"2024-09-11T05:06:45.379926+0000","last_fullsized":"2024-09-11T05:06:45.379926+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11861","reported_seq":11973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.145967+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:49.145967+0000","last_peered":"2024-09-11T05:06:49.145967+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:49.145967+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5985440,"num_objects":731,"num_object_clones":0,"num_object_copies":2924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11859,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11734","reported_seq":11857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.201449+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:49.201449+0000","last_peered":"2024-09-11T05:06:49.201449+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:49.201449+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5526688,"num_objects":675,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11732,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12000","reported_seq":12125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:46.156356+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:46.156356+0000","last_peered":"2024-09-11T05:06:46.156356+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:46.156356+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6362272,"num_objects":777,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":97507224,"num_objects":11908,"num_object_clones":0,"num_object_copies":47632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162142,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":195620864,"data_stored":195620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160698,"ondisk_log_size":160698,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754952,"kb_used_data":47160,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93616888,"statfs":{"total":96636764160,"available":95863693312,"internally_reserved":0,"allocated":48291840,"data_stored":48093331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736088,"kb_used_data":46844,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93635752,"statfs":{"total":96636764160,"available":95883010048,"internally_reserved":0,"allocated":47968256,"data_stored":47769747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582820,"kb_used_data":50004,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93789020,"statfs":{"total":96636764160,"available":96039956480,"internally_reserved":0,"allocated":51204096,"data_stored":51005587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545582802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,4,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730988,"kb_used_data":49628,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93640852,"statfs":{"total":96636764160,"available":95888232448,"internally_reserved":0,"allocated":50819072,"data_stored":50620563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49807360,"data_stored":49807360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50733056,"data_stored":50733056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":47570944,"data_stored":47570944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47509504,"data_stored":47509504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:50.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:50.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11781", 'reported_seq': 11880, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.159403+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:49.159403+0000', 'last_peered': '2024-09-11T05:06:49.159403+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:49.159403+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6004736, 'num_objects': 733, 'num_object_clones': 0, 'num_object_copies': 2932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11781, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1558}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:50.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11983", 'reported_seq': 12101, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:46.515338+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:46.515338+0000', 'last_peered': '2024-09-11T05:06:46.515338+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:46.515338+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6182048, 'num_objects': 755, 'num_object_clones': 0, 'num_object_copies': 3020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11981, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11467", 'reported_seq': 11577, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.035162+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:49.035162+0000', 'last_peered': '2024-09-11T05:06:49.035162+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:49.035162+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5675624, 'num_objects': 694, 'num_object_clones': 0, 'num_object_copies': 2776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 694, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11466, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1517}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11680", 'reported_seq': 11999, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:46.488812+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:06:46.488812+0000', 'last_peered': '2024-09-11T05:06:46.488812+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:46.488812+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6026400, 'num_objects': 736, 'num_object_clones': 0, 'num_object_copies': 2944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 122, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11826", 'reported_seq': 11954, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.146267+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:49.146267+0000', 'last_peered': '2024-09-11T05:06:49.146267+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:49.146267+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5772448, 'num_objects': 705, 'num_object_clones': 0, 'num_object_copies': 2820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11824, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1516}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11917", 'reported_seq': 12027, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.302053+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:49.302053+0000', 'last_peered': '2024-09-11T05:06:49.302053+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:49.302053+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5922816, 'num_objects': 723, 'num_object_clones': 0, 'num_object_copies': 2892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11917, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1571}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:50.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11945", 'reported_seq': 12044, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:46.369637+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:46.369637+0000', 'last_peered': '2024-09-11T05:06:46.369637+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:46.369637+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6427808, 'num_objects': 785, 'num_object_clones': 0, 'num_object_copies': 3140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11943, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1522}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12029", 'reported_seq': 12157, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.057935+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:49.057935+0000', 'last_peered': '2024-09-11T05:06:49.057935+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:49.057935+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6211904, 'num_objects': 759, 'num_object_clones': 0, 'num_object_copies': 3036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12026, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1607}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11835", 'reported_seq': 11952, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.033379+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:49.033379+0000', 'last_peered': '2024-09-11T05:06:49.033379+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:49.033379+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5674144, 'num_objects': 693, 'num_object_clones': 0, 'num_object_copies': 2772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11833, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1613}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11917", 'reported_seq': 12039, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.239646+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:49.239646+0000', 'last_peered': '2024-09-11T05:06:49.239646+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:49.239646+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6024944, 'num_objects': 736, 'num_object_clones': 0, 'num_object_copies': 2944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11914, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1562}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11861", 'reported_seq': 11973, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.145967+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:49.145967+0000', 'last_peered': '2024-09-11T05:06:49.145967+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:49.145967+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5985440, 'num_objects': 731, 'num_object_clones': 0, 'num_object_copies': 2924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11859, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1533}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:50.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:50.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11734", 'reported_seq': 11857, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:49.201449+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:49.201449+0000', 'last_peered': '2024-09-11T05:06:49.201449+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:49.201449+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5526688, 'num_objects': 675, 'num_object_clones': 0, 'num_object_copies': 2700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11732, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1628}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:50.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:50.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12000", 'reported_seq': 12125, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:46.156356+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:46.156356+0000', 'last_peered': '2024-09-11T05:06:46.156356+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:46.156356+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6362272, 'num_objects': 777, 'num_object_clones': 0, 'num_object_copies': 3108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:50.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:51.417 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:51.417 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29588/47632 objects degraded (62.118%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":738,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11908,"data_bytes":97507224,"bytes_used":2872164352,"bytes_avail":383674892288,"bytes_total":386547056640,"degraded_objects":29588,"degraded_total":47632,"degraded_ratio":0.62117903930131002},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:51.442 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:06:51.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:52.100 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:52.102 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:52.131 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-11T05:06:51.905375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93796248,"num_objects":11455,"num_object_clones":0,"num_object_copies":45820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11455,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162595,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160751,"ondisk_log_size":160751,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804820,"kb_used_data":185272,"kb_used_omap":14,"kb_used_meta":2619441,"kb_avail":374682540,"statfs":{"total":386547056640,"available":383674920960,"internally_reserved":0,"allocated":189718528,"data_stored":188924492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2682308150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,2,10,13,16,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19528272,"num_objects":-2384,"num_object_clones":0,"num_object_copies":-9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9469,"num_write_kb":-50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005906"},"pg_stats":[{"pgid":"1.f","version":"37'11781","reported_seq":11880,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.159403+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:49.159403+0000","last_peered":"2024-09-11T05:06:49.159403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:49.159403+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12040","reported_seq":12160,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.451256+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:51.451256+0000","last_peered":"2024-09-11T05:06:51.451256+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:51.451256+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5715104,"num_objects":698,"num_object_clones":0,"num_object_copies":2792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11467","reported_seq":11577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.035162+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:49.035162+0000","last_peered":"2024-09-11T05:06:49.035162+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:49.035162+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5675624,"num_objects":694,"num_object_clones":0,"num_object_copies":2776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11466,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11747","reported_seq":12155,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.579950+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:51.579950+0000","last_peered":"2024-09-11T05:06:51.579950+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:51.579950+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5477536,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11826","reported_seq":11954,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.146267+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:49.146267+0000","last_peered":"2024-09-11T05:06:49.146267+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:49.146267+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5772448,"num_objects":705,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11917","reported_seq":12027,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.302053+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:49.302053+0000","last_peered":"2024-09-11T05:06:49.302053+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:49.302053+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":723,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12005","reported_seq":12104,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.515095+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:51.515095+0000","last_peered":"2024-09-11T05:06:51.515095+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:51.515095+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5936288,"num_objects":725,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1582}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12096","reported_seq":16956,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:50.397116+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:50.397116+0000","last_peered":"2024-09-11T05:06:50.397116+0000","last_clean":"2024-09-11T05:06:50.397116+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:50.397116+0000","last_undegraded":"2024-09-11T05:06:50.397116+0000","last_fullsized":"2024-09-11T05:06:50.397116+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5955584,"num_objects":727,"num_object_clones":0,"num_object_copies":2908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12029","reported_seq":12157,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.057935+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:49.057935+0000","last_peered":"2024-09-11T05:06:49.057935+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:49.057935+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6211904,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12026,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11835","reported_seq":11952,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.033379+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:49.033379+0000","last_peered":"2024-09-11T05:06:49.033379+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:49.033379+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5674144,"num_objects":693,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11917","reported_seq":12039,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.239646+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:49.239646+0000","last_peered":"2024-09-11T05:06:49.239646+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:49.239646+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6024944,"num_objects":736,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11664","reported_seq":16270,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:50.459353+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:50.459353+0000","last_peered":"2024-09-11T05:06:50.459353+0000","last_clean":"2024-09-11T05:06:50.459353+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:50.459353+0000","last_undegraded":"2024-09-11T05:06:50.459353+0000","last_fullsized":"2024-09-11T05:06:50.459353+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11972","reported_seq":16754,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:50.376894+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:06:50.376894+0000","last_peered":"2024-09-11T05:06:50.376894+0000","last_clean":"2024-09-11T05:06:50.376894+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:50.376894+0000","last_undegraded":"2024-09-11T05:06:50.376894+0000","last_fullsized":"2024-09-11T05:06:50.376894+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6144000,"num_objects":750,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11861","reported_seq":11973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.145967+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:49.145967+0000","last_peered":"2024-09-11T05:06:49.145967+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:49.145967+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5985440,"num_objects":731,"num_object_clones":0,"num_object_copies":2924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11859,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11734","reported_seq":11857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.201449+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:49.201449+0000","last_peered":"2024-09-11T05:06:49.201449+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:49.201449+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5526688,"num_objects":675,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11732,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12060","reported_seq":12183,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.302902+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:51.302902+0000","last_peered":"2024-09-11T05:06:51.302902+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:51.302902+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5870752,"num_objects":717,"num_object_clones":0,"num_object_copies":2868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12058,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1571}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":93796248,"num_objects":11455,"num_object_clones":0,"num_object_copies":45820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11455,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162595,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187604992,"data_stored":187604992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160751,"ondisk_log_size":160751,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754952,"kb_used_data":47160,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93616888,"statfs":{"total":96636764160,"available":95863693312,"internally_reserved":0,"allocated":48291840,"data_stored":48093331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736088,"kb_used_data":46844,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93635752,"statfs":{"total":96636764160,"available":95883010048,"internally_reserved":0,"allocated":47968256,"data_stored":47769747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583136,"kb_used_data":46148,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93788704,"statfs":{"total":96636764160,"available":96039632896,"internally_reserved":0,"allocated":47255552,"data_stored":47057043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549842642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,3,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730644,"kb_used_data":45120,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93641196,"statfs":{"total":96636764160,"available":95888584704,"internally_reserved":0,"allocated":46202880,"data_stored":46004371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45813760,"data_stored":45813760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46710784,"data_stored":46710784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":47570944,"data_stored":47570944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47509504,"data_stored":47509504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:55.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:55.735 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:55.736 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:55.767 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-11T05:06:53.906202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93796248,"num_objects":11455,"num_object_clones":0,"num_object_copies":45820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11455,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162595,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160751,"ondisk_log_size":160751,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804820,"kb_used_data":185272,"kb_used_omap":14,"kb_used_meta":2619441,"kb_avail":374682540,"statfs":{"total":386547056640,"available":383674920960,"internally_reserved":0,"allocated":189718528,"data_stored":188924492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2682308150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,2,10,13,16,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16292432,"num_objects":-1989,"num_object_clones":0,"num_object_copies":-7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9864,"num_write_kb":-50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005534"},"pg_stats":[{"pgid":"1.f","version":"37'11781","reported_seq":11880,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.159403+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:49.159403+0000","last_peered":"2024-09-11T05:06:49.159403+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:49.159403+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12040","reported_seq":12160,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.451256+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:51.451256+0000","last_peered":"2024-09-11T05:06:51.451256+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:51.451256+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5715104,"num_objects":698,"num_object_clones":0,"num_object_copies":2792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11467","reported_seq":11577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.035162+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:49.035162+0000","last_peered":"2024-09-11T05:06:49.035162+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:49.035162+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5675624,"num_objects":694,"num_object_clones":0,"num_object_copies":2776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11466,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11747","reported_seq":12155,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.579950+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:51.579950+0000","last_peered":"2024-09-11T05:06:51.579950+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:51.579950+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5477536,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11826","reported_seq":11954,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.146267+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:49.146267+0000","last_peered":"2024-09-11T05:06:49.146267+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:49.146267+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5772448,"num_objects":705,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11917","reported_seq":12027,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.302053+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:49.302053+0000","last_peered":"2024-09-11T05:06:49.302053+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:49.302053+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":723,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12005","reported_seq":12104,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.515095+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:51.515095+0000","last_peered":"2024-09-11T05:06:51.515095+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:51.515095+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5936288,"num_objects":725,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1582}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12096","reported_seq":16956,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:50.397116+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:50.397116+0000","last_peered":"2024-09-11T05:06:50.397116+0000","last_clean":"2024-09-11T05:06:50.397116+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:50.397116+0000","last_undegraded":"2024-09-11T05:06:50.397116+0000","last_fullsized":"2024-09-11T05:06:50.397116+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5955584,"num_objects":727,"num_object_clones":0,"num_object_copies":2908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12029","reported_seq":12157,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.057935+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:49.057935+0000","last_peered":"2024-09-11T05:06:49.057935+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:49.057935+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6211904,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12026,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11835","reported_seq":11952,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.033379+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:49.033379+0000","last_peered":"2024-09-11T05:06:49.033379+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:49.033379+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5674144,"num_objects":693,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11917","reported_seq":12039,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.239646+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:49.239646+0000","last_peered":"2024-09-11T05:06:49.239646+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:49.239646+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6024944,"num_objects":736,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11664","reported_seq":16270,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:50.459353+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:50.459353+0000","last_peered":"2024-09-11T05:06:50.459353+0000","last_clean":"2024-09-11T05:06:50.459353+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:50.459353+0000","last_undegraded":"2024-09-11T05:06:50.459353+0000","last_fullsized":"2024-09-11T05:06:50.459353+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11972","reported_seq":16754,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:50.376894+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:06:50.376894+0000","last_peered":"2024-09-11T05:06:50.376894+0000","last_clean":"2024-09-11T05:06:50.376894+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:50.376894+0000","last_undegraded":"2024-09-11T05:06:50.376894+0000","last_fullsized":"2024-09-11T05:06:50.376894+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6144000,"num_objects":750,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11861","reported_seq":11973,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.145967+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:49.145967+0000","last_peered":"2024-09-11T05:06:49.145967+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:49.145967+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5985440,"num_objects":731,"num_object_clones":0,"num_object_copies":2924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11859,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11734","reported_seq":11857,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:49.201449+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:49.201449+0000","last_peered":"2024-09-11T05:06:49.201449+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:49.201449+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5526688,"num_objects":675,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11732,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12060","reported_seq":12183,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.302902+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:51.302902+0000","last_peered":"2024-09-11T05:06:51.302902+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:51.302902+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5870752,"num_objects":717,"num_object_clones":0,"num_object_copies":2868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12058,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1571}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":93796248,"num_objects":11455,"num_object_clones":0,"num_object_copies":45820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11455,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162595,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187604992,"data_stored":187604992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160751,"ondisk_log_size":160751,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754952,"kb_used_data":47160,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93616888,"statfs":{"total":96636764160,"available":95863693312,"internally_reserved":0,"allocated":48291840,"data_stored":48093331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736088,"kb_used_data":46844,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93635752,"statfs":{"total":96636764160,"available":95883010048,"internally_reserved":0,"allocated":47968256,"data_stored":47769747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583136,"kb_used_data":46148,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93788704,"statfs":{"total":96636764160,"available":96039632896,"internally_reserved":0,"allocated":47255552,"data_stored":47057043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549842642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,3,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730644,"kb_used_data":45120,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93641196,"statfs":{"total":96636764160,"available":95888584704,"internally_reserved":0,"allocated":46202880,"data_stored":46004371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45813760,"data_stored":45813760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46710784,"data_stored":46710784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":47570944,"data_stored":47570944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47509504,"data_stored":47509504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:55.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:56.442 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:56.443 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:56.471 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-11T05:06:55.907002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88020888,"num_objects":10750,"num_object_clones":0,"num_object_copies":43000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163300,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789268,"kb_used_data":173852,"kb_used_omap":14,"kb_used_meta":2615281,"kb_avail":374698092,"statfs":{"total":386547056640,"available":383690846208,"internally_reserved":0,"allocated":178024448,"data_stored":177230412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2678048310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,3,6,6,10,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22067792,"num_objects":-2694,"num_object_clones":0,"num_object_copies":-10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9159,"num_write_kb":-50887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005009"},"pg_stats":[{"pgid":"1.f","version":"37'11832","reported_seq":11932,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.347203+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:54.347203+0000","last_peered":"2024-09-11T05:06:54.347203+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:54.347203+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5586944,"num_objects":682,"num_object_clones":0,"num_object_copies":2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12040","reported_seq":12160,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.451256+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:51.451256+0000","last_peered":"2024-09-11T05:06:51.451256+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:51.451256+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5715104,"num_objects":698,"num_object_clones":0,"num_object_copies":2792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11526","reported_seq":11635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.167506+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:54.167506+0000","last_peered":"2024-09-11T05:06:54.167506+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:54.167506+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5192296,"num_objects":635,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11525,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11747","reported_seq":12155,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.579950+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:51.579950+0000","last_peered":"2024-09-11T05:06:51.579950+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:51.579950+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5477536,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11891","reported_seq":12019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.265440+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:54.265440+0000","last_peered":"2024-09-11T05:06:54.265440+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:54.265440+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5239968,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11889,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1581}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11975","reported_seq":12084,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.252181+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:54.252181+0000","last_peered":"2024-09-11T05:06:54.252181+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:54.252181+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5447680,"num_objects":665,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1629}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12005","reported_seq":12104,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.515095+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:51.515095+0000","last_peered":"2024-09-11T05:06:51.515095+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:51.515095+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5936288,"num_objects":725,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1582}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12155","reported_seq":17017,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.575572+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:55.575572+0000","last_peered":"2024-09-11T05:06:55.575572+0000","last_clean":"2024-09-11T05:06:55.575572+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:55.575572+0000","last_undegraded":"2024-09-11T05:06:55.575572+0000","last_fullsized":"2024-09-11T05:06:55.575572+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5472256,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12075","reported_seq":12203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.246364+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:54.246364+0000","last_peered":"2024-09-11T05:06:54.246364+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:54.246364+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5835072,"num_objects":713,"num_object_clones":0,"num_object_copies":2852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1653}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11897","reported_seq":12012,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.296707+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:54.296707+0000","last_peered":"2024-09-11T05:06:54.296707+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:54.296707+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5166240,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11895,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1675}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11973","reported_seq":12095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.276176+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:54.276176+0000","last_peered":"2024-09-11T05:06:54.276176+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:54.276176+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5566192,"num_objects":680,"num_object_clones":0,"num_object_copies":2720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11736","reported_seq":16342,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.708580+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:55.708580+0000","last_peered":"2024-09-11T05:06:55.708580+0000","last_clean":"2024-09-11T05:06:55.708580+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:55.708580+0000","last_undegraded":"2024-09-11T05:06:55.708580+0000","last_fullsized":"2024-09-11T05:06:55.708580+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":648,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5546,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12033","reported_seq":16815,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.342535+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:06:55.342535+0000","last_peered":"2024-09-11T05:06:55.342535+0000","last_clean":"2024-09-11T05:06:55.342535+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:55.342535+0000","last_undegraded":"2024-09-11T05:06:55.342535+0000","last_fullsized":"2024-09-11T05:06:55.342535+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5644288,"num_objects":689,"num_object_clones":0,"num_object_copies":2756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11919","reported_seq":12031,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.181568+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:54.181568+0000","last_peered":"2024-09-11T05:06:54.181568+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:54.181568+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5510304,"num_objects":673,"num_object_clones":0,"num_object_copies":2692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1591}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11792","reported_seq":11916,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.248411+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:54.248411+0000","last_peered":"2024-09-11T05:06:54.248411+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:54.248411+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5051552,"num_objects":617,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12060","reported_seq":12183,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:51.302902+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:51.302902+0000","last_peered":"2024-09-11T05:06:51.302902+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:51.302902+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5870752,"num_objects":717,"num_object_clones":0,"num_object_copies":2868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12058,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1571}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":88020888,"num_objects":10750,"num_object_clones":0,"num_object_copies":43000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163300,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":175607808,"data_stored":175607808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738680,"kb_used_data":43348,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93633160,"statfs":{"total":96636764160,"available":95880355840,"internally_reserved":0,"allocated":44388352,"data_stored":44189843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":711978704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736492,"kb_used_data":43072,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93635348,"statfs":{"total":96636764160,"available":95882596352,"internally_reserved":0,"allocated":44105728,"data_stored":43907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,0,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583452,"kb_used_data":42312,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93788388,"statfs":{"total":96636764160,"available":96039309312,"internally_reserved":0,"allocated":43327488,"data_stored":43128979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554102482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730644,"kb_used_data":45120,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93641196,"statfs":{"total":96636764160,"available":95888584704,"internally_reserved":0,"allocated":46202880,"data_stored":46004371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45813760,"data_stored":45813760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42598400,"data_stored":42598400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":43585536,"data_stored":43585536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":43610112,"data_stored":43610112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:06:56.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:06:56.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11832", 'reported_seq': 11932, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.347203+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:54.347203+0000', 'last_peered': '2024-09-11T05:06:54.347203+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:54.347203+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5586944, 'num_objects': 682, 'num_object_clones': 0, 'num_object_copies': 2728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11832, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1609}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:56.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:06:56.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12040", 'reported_seq': 12160, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:51.451256+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:06:51.451256+0000', 'last_peered': '2024-09-11T05:06:51.451256+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:06:51.451256+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5715104, 'num_objects': 698, 'num_object_clones': 0, 'num_object_copies': 2792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12038, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1560}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:56.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:06:56.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11526", 'reported_seq': 11635, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.167506+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:54.167506+0000', 'last_peered': '2024-09-11T05:06:54.167506+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:54.167506+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5192296, 'num_objects': 635, 'num_object_clones': 0, 'num_object_copies': 2540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 635, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11525, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1576}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:56.473 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:06:56.473 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11747", 'reported_seq': 12155, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:51.579950+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:06:51.579950+0000', 'last_peered': '2024-09-11T05:06:51.579950+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:06:51.579950+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5477536, 'num_objects': 669, 'num_object_clones': 0, 'num_object_copies': 2676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11745, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 166, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1447}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11891", 'reported_seq': 12019, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.265440+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:54.265440+0000', 'last_peered': '2024-09-11T05:06:54.265440+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:54.265440+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5239968, 'num_objects': 640, 'num_object_clones': 0, 'num_object_copies': 2560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11889, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1581}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11975", 'reported_seq': 12084, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.252181+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:54.252181+0000', 'last_peered': '2024-09-11T05:06:54.252181+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:54.252181+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5447680, 'num_objects': 665, 'num_object_clones': 0, 'num_object_copies': 2660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11975, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1629}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12005", 'reported_seq': 12104, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:51.515095+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:06:51.515095+0000', 'last_peered': '2024-09-11T05:06:51.515095+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:06:51.515095+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5936288, 'num_objects': 725, 'num_object_clones': 0, 'num_object_copies': 2900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12003, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1582}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12075", 'reported_seq': 12203, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.246364+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:54.246364+0000', 'last_peered': '2024-09-11T05:06:54.246364+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:54.246364+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5835072, 'num_objects': 713, 'num_object_clones': 0, 'num_object_copies': 2852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12072, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1653}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11897", 'reported_seq': 12012, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.296707+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:54.296707+0000', 'last_peered': '2024-09-11T05:06:54.296707+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:54.296707+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5166240, 'num_objects': 631, 'num_object_clones': 0, 'num_object_copies': 2524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11895, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1675}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:56.474 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:06:56.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11973", 'reported_seq': 12095, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.276176+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:54.276176+0000', 'last_peered': '2024-09-11T05:06:54.276176+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:54.276176+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5566192, 'num_objects': 680, 'num_object_clones': 0, 'num_object_copies': 2720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11970, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1618}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:56.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:06:56.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11919", 'reported_seq': 12031, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.181568+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:54.181568+0000', 'last_peered': '2024-09-11T05:06:54.181568+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:54.181568+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5510304, 'num_objects': 673, 'num_object_clones': 0, 'num_object_copies': 2692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11917, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:06:56.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:06:56.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11792", 'reported_seq': 11916, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:54.248411+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:54.248411+0000', 'last_peered': '2024-09-11T05:06:54.248411+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:54.248411+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5051552, 'num_objects': 617, 'num_object_clones': 0, 'num_object_copies': 2468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11790, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:06:56.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:06:56.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12060", 'reported_seq': 12183, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:51.302902+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:06:51.302902+0000', 'last_peered': '2024-09-11T05:06:51.302902+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:06:51.302902+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5870752, 'num_objects': 717, 'num_object_clones': 0, 'num_object_copies': 2868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12058, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1571}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:06:56.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:06:57.338 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:57.338 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29544/43000 objects degraded (68.707%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":744,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10750,"data_bytes":88020888,"bytes_used":2856210432,"bytes_avail":383690846208,"bytes_total":386547056640,"degraded_objects":29544,"degraded_total":43000,"degraded_ratio":0.68706976744186055},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:06:57.366 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:06:57.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:06:58.028 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:06:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:06:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-11T05:06:57.907863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":86095768,"num_objects":10515,"num_object_clones":0,"num_object_copies":42060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163535,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160891,"ondisk_log_size":160891,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789640,"kb_used_data":170060,"kb_used_omap":14,"kb_used_meta":2619441,"kb_avail":374697720,"statfs":{"total":386547056640,"available":383690465280,"internally_reserved":0,"allocated":174141440,"data_stored":173347404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2682308150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,3,4,12,11,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17760256,"num_objects":-2168,"num_object_clones":0,"num_object_copies":-8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005074"},"pg_stats":[{"pgid":"1.f","version":"37'11832","reported_seq":11932,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.347203+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:54.347203+0000","last_peered":"2024-09-11T05:06:54.347203+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:54.347203+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5586944,"num_objects":682,"num_object_clones":0,"num_object_copies":2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12095","reported_seq":12215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:56.569033+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:56.569033+0000","last_peered":"2024-09-11T05:06:56.569033+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:56.569033+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5264544,"num_objects":643,"num_object_clones":0,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12093,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11526","reported_seq":11635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.167506+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:54.167506+0000","last_peered":"2024-09-11T05:06:54.167506+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:54.167506+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5192296,"num_objects":635,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11525,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11806","reported_seq":12310,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:56.569854+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:56.569854+0000","last_peered":"2024-09-11T05:06:56.569854+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:56.569854+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4994208,"num_objects":610,"num_object_clones":0,"num_object_copies":2440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11804,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1458}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11891","reported_seq":12019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.265440+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:54.265440+0000","last_peered":"2024-09-11T05:06:54.265440+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:54.265440+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5239968,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11889,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1581}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11975","reported_seq":12084,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.252181+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:54.252181+0000","last_peered":"2024-09-11T05:06:54.252181+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:54.252181+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5447680,"num_objects":665,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1629}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12065","reported_seq":12164,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:56.524403+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:56.524403+0000","last_peered":"2024-09-11T05:06:56.524403+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:56.524403+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5444768,"num_objects":665,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12063,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1642}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12155","reported_seq":17017,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.575572+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:55.575572+0000","last_peered":"2024-09-11T05:06:55.575572+0000","last_clean":"2024-09-11T05:06:55.575572+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:55.575572+0000","last_undegraded":"2024-09-11T05:06:55.575572+0000","last_fullsized":"2024-09-11T05:06:55.575572+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5472256,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12075","reported_seq":12203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.246364+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:54.246364+0000","last_peered":"2024-09-11T05:06:54.246364+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:54.246364+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5835072,"num_objects":713,"num_object_clones":0,"num_object_copies":2852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1653}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11897","reported_seq":12012,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.296707+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:54.296707+0000","last_peered":"2024-09-11T05:06:54.296707+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:54.296707+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5166240,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11895,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1675}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11973","reported_seq":12095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.276176+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:54.276176+0000","last_peered":"2024-09-11T05:06:54.276176+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:54.276176+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5566192,"num_objects":680,"num_object_clones":0,"num_object_copies":2720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11736","reported_seq":16342,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.708580+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:55.708580+0000","last_peered":"2024-09-11T05:06:55.708580+0000","last_clean":"2024-09-11T05:06:55.708580+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:55.708580+0000","last_undegraded":"2024-09-11T05:06:55.708580+0000","last_fullsized":"2024-09-11T05:06:55.708580+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":648,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5546,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12033","reported_seq":16815,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.342535+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:06:55.342535+0000","last_peered":"2024-09-11T05:06:55.342535+0000","last_clean":"2024-09-11T05:06:55.342535+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:55.342535+0000","last_undegraded":"2024-09-11T05:06:55.342535+0000","last_fullsized":"2024-09-11T05:06:55.342535+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5644288,"num_objects":689,"num_object_clones":0,"num_object_copies":2756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11919","reported_seq":12031,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.181568+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:54.181568+0000","last_peered":"2024-09-11T05:06:54.181568+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:54.181568+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5510304,"num_objects":673,"num_object_clones":0,"num_object_copies":2692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1591}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11792","reported_seq":11916,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:54.248411+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:54.248411+0000","last_peered":"2024-09-11T05:06:54.248411+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:54.248411+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5051552,"num_objects":617,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1686}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12121","reported_seq":12246,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:56.517226+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:56.517226+0000","last_peered":"2024-09-11T05:06:56.517226+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:56.517226+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5371040,"num_objects":656,"num_object_clones":0,"num_object_copies":2624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12119,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":86095768,"num_objects":10515,"num_object_clones":0,"num_object_copies":42060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163535,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171655168,"data_stored":171655168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160891,"ondisk_log_size":160891,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738680,"kb_used_data":43348,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93633160,"statfs":{"total":96636764160,"available":95880355840,"internally_reserved":0,"allocated":44388352,"data_stored":44189843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":711978704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736492,"kb_used_data":43072,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93635348,"statfs":{"total":96636764160,"available":95882596352,"internally_reserved":0,"allocated":44105728,"data_stored":43907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,0,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583452,"kb_used_data":42312,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93788388,"statfs":{"total":96636764160,"available":96039309312,"internally_reserved":0,"allocated":43327488,"data_stored":43128979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554102482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731016,"kb_used_data":41328,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93640824,"statfs":{"total":96636764160,"available":95888203776,"internally_reserved":0,"allocated":42319872,"data_stored":42121363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706211535},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41861120,"data_stored":41861120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42598400,"data_stored":42598400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":43585536,"data_stored":43585536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":43610112,"data_stored":43610112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:01.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:01.677 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:01.678 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:01.696 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-11T05:06:59.908700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81581976,"num_objects":9964,"num_object_clones":0,"num_object_copies":39856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164086,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773844,"kb_used_data":162624,"kb_used_omap":14,"kb_used_meta":2611121,"kb_avail":374713516,"statfs":{"total":386547056640,"available":383706640384,"internally_reserved":0,"allocated":166526976,"data_stored":165732940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2673788470},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,6,20,20,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20307968,"num_objects":-2479,"num_object_clones":0,"num_object_copies":-9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2479,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004953"},"pg_stats":[{"pgid":"1.f","version":"37'11891","reported_seq":11990,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.227810+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:59.227810+0000","last_peered":"2024-09-11T05:06:59.227810+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:59.227810+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5103616,"num_objects":623,"num_object_clones":0,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12095","reported_seq":12215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:56.569033+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:06:56.569033+0000","last_peered":"2024-09-11T05:06:56.569033+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:06:56.569033+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5264544,"num_objects":643,"num_object_clones":0,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12093,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11590","reported_seq":11700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.280896+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:59.280896+0000","last_peered":"2024-09-11T05:06:59.280896+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:59.280896+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4668008,"num_objects":571,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11589,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11806","reported_seq":12310,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:56.569854+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:06:56.569854+0000","last_peered":"2024-09-11T05:06:56.569854+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:06:56.569854+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4994208,"num_objects":610,"num_object_clones":0,"num_object_copies":2440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11804,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1458}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11943","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.261148+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:59.261148+0000","last_peered":"2024-09-11T05:06:59.261148+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:59.261148+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4813984,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11941,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12025","reported_seq":12136,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.292021+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:59.292021+0000","last_peered":"2024-09-11T05:06:59.292021+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:59.292021+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5038080,"num_objects":615,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12025,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12065","reported_seq":12164,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:56.524403+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:06:56.524403+0000","last_peered":"2024-09-11T05:06:56.524403+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:06:56.524403+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5444768,"num_objects":665,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12063,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1642}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12155","reported_seq":17017,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.575572+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:06:55.575572+0000","last_peered":"2024-09-11T05:06:55.575572+0000","last_clean":"2024-09-11T05:06:55.575572+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:06:55.575572+0000","last_undegraded":"2024-09-11T05:06:55.575572+0000","last_fullsized":"2024-09-11T05:06:55.575572+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5472256,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12160","reported_seq":12285,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.421782+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:59.421782+0000","last_peered":"2024-09-11T05:06:59.421782+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:59.421782+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5138752,"num_objects":628,"num_object_clones":0,"num_object_copies":2512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11965","reported_seq":12081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.389884+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:59.389884+0000","last_peered":"2024-09-11T05:06:59.389884+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:59.389884+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4609184,"num_objects":563,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12015","reported_seq":12137,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.290794+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:59.290794+0000","last_peered":"2024-09-11T05:06:59.290794+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:59.290794+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5222128,"num_objects":638,"num_object_clones":0,"num_object_copies":2552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1660}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11736","reported_seq":16342,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.708580+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:06:55.708580+0000","last_peered":"2024-09-11T05:06:55.708580+0000","last_clean":"2024-09-11T05:06:55.708580+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:06:55.708580+0000","last_undegraded":"2024-09-11T05:06:55.708580+0000","last_fullsized":"2024-09-11T05:06:55.708580+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":648,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5546,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12033","reported_seq":16815,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:06:55.342535+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:06:55.342535+0000","last_peered":"2024-09-11T05:06:55.342535+0000","last_clean":"2024-09-11T05:06:55.342535+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:06:55.342535+0000","last_undegraded":"2024-09-11T05:06:55.342535+0000","last_fullsized":"2024-09-11T05:06:55.342535+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5644288,"num_objects":689,"num_object_clones":0,"num_object_copies":2756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11983","reported_seq":12094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.129429+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:59.129429+0000","last_peered":"2024-09-11T05:06:59.129429+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:59.129429+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986016,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1655}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11859","reported_seq":11983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.329244+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:59.329244+0000","last_peered":"2024-09-11T05:06:59.329244+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:59.329244+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4502688,"num_objects":550,"num_object_clones":0,"num_object_copies":2200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11857,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1753}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12121","reported_seq":12246,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:56.517226+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:06:56.517226+0000","last_peered":"2024-09-11T05:06:56.517226+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:06:56.517226+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5371040,"num_objects":656,"num_object_clones":0,"num_object_copies":2624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12119,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81581976,"num_objects":9964,"num_object_clones":0,"num_object_copies":39856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164086,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163827712,"data_stored":163827712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739092,"kb_used_data":39612,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93632748,"statfs":{"total":96636764160,"available":95879933952,"internally_reserved":0,"allocated":40562688,"data_stored":40364179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720284,"kb_used_data":39372,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93651556,"statfs":{"total":96636764160,"available":95899193344,"internally_reserved":0,"allocated":40316928,"data_stored":40118419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,9,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583452,"kb_used_data":42312,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93788388,"statfs":{"total":96636764160,"available":96039309312,"internally_reserved":0,"allocated":43327488,"data_stored":43128979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":554102482},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731016,"kb_used_data":41328,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93640824,"statfs":{"total":96636764160,"available":95888203776,"internally_reserved":0,"allocated":42319872,"data_stored":42121363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706211535},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41861120,"data_stored":41861120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42598400,"data_stored":42598400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39677952,"data_stored":39677952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39690240,"data_stored":39690240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:01.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:02.344 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:02.344 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:02.371 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-11T05:07:01.909630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":78133144,"num_objects":9543,"num_object_clones":0,"num_object_copies":38172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164507,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757944,"kb_used_data":155040,"kb_used_omap":14,"kb_used_meta":2602801,"kb_avail":374729416,"statfs":{"total":386547056640,"available":383722921984,"internally_reserved":0,"allocated":158760960,"data_stored":157966924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2665268790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,11,7,20,23,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19374080,"num_objects":-2365,"num_object_clones":0,"num_object_copies":-9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005100"},"pg_stats":[{"pgid":"1.f","version":"37'11891","reported_seq":11990,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.227810+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:59.227810+0000","last_peered":"2024-09-11T05:06:59.227810+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:59.227810+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5103616,"num_objects":623,"num_object_clones":0,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12152","reported_seq":12272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.595907+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:01.595907+0000","last_peered":"2024-09-11T05:07:01.595907+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:01.595907+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4797600,"num_objects":586,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11590","reported_seq":11700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.280896+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:59.280896+0000","last_peered":"2024-09-11T05:06:59.280896+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:59.280896+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4668008,"num_objects":571,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11589,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11846","reported_seq":12447,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.716519+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:01.716519+0000","last_peered":"2024-09-11T05:07:01.716519+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:01.716519+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4666528,"num_objects":570,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11943","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.261148+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:59.261148+0000","last_peered":"2024-09-11T05:06:59.261148+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:59.261148+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4813984,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11941,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12025","reported_seq":12136,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.292021+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:59.292021+0000","last_peered":"2024-09-11T05:06:59.292021+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:59.292021+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5038080,"num_objects":615,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12025,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12122","reported_seq":12221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.304789+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:01.304789+0000","last_peered":"2024-09-11T05:07:01.304789+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:01.304789+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4977824,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12120,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1699}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12225","reported_seq":17087,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.772663+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:00.772663+0000","last_peered":"2024-09-11T05:07:00.772663+0000","last_clean":"2024-09-11T05:07:00.772663+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:00.772663+0000","last_undegraded":"2024-09-11T05:07:00.772663+0000","last_fullsized":"2024-09-11T05:07:00.772663+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4898816,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12160","reported_seq":12285,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.421782+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:59.421782+0000","last_peered":"2024-09-11T05:06:59.421782+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:59.421782+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5138752,"num_objects":628,"num_object_clones":0,"num_object_copies":2512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11965","reported_seq":12081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.389884+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:59.389884+0000","last_peered":"2024-09-11T05:06:59.389884+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:59.389884+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4609184,"num_objects":563,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12015","reported_seq":12137,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.290794+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:59.290794+0000","last_peered":"2024-09-11T05:06:59.290794+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:59.290794+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5222128,"num_objects":638,"num_object_clones":0,"num_object_copies":2552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1660}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11807","reported_seq":16413,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.601452+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:00.601452+0000","last_peered":"2024-09-11T05:07:00.601452+0000","last_clean":"2024-09-11T05:07:00.601452+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:00.601452+0000","last_undegraded":"2024-09-11T05:07:00.601452+0000","last_fullsized":"2024-09-11T05:07:00.601452+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4726784,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5617,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12100","reported_seq":16882,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.637744+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:00.637744+0000","last_peered":"2024-09-11T05:07:00.637744+0000","last_clean":"2024-09-11T05:07:00.637744+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:00.637744+0000","last_undegraded":"2024-09-11T05:07:00.637744+0000","last_fullsized":"2024-09-11T05:07:00.637744+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5095424,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11983","reported_seq":12094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.129429+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:59.129429+0000","last_peered":"2024-09-11T05:06:59.129429+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:59.129429+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986016,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1655}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11859","reported_seq":11983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.329244+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:59.329244+0000","last_peered":"2024-09-11T05:06:59.329244+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:59.329244+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4502688,"num_objects":550,"num_object_clones":0,"num_object_copies":2200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11857,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1753}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12180","reported_seq":12305,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.571185+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:01.571185+0000","last_peered":"2024-09-11T05:07:01.571185+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:01.571185+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4887712,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":78133144,"num_objects":9543,"num_object_clones":0,"num_object_copies":38172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164507,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155860992,"data_stored":155860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739092,"kb_used_data":39612,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93632748,"statfs":{"total":96636764160,"available":95879933952,"internally_reserved":0,"allocated":40562688,"data_stored":40364179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720284,"kb_used_data":39372,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93651556,"statfs":{"total":96636764160,"available":95899193344,"internally_reserved":0,"allocated":40316928,"data_stored":40118419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,9,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583848,"kb_used_data":38540,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93787992,"statfs":{"total":96636764160,"available":96038903808,"internally_reserved":0,"allocated":39464960,"data_stored":39266451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558362322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,6,9,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714720,"kb_used_data":37516,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93657120,"statfs":{"total":96636764160,"available":95904890880,"internally_reserved":0,"allocated":38416384,"data_stored":38217875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693432015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37879808,"data_stored":37879808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38612992,"data_stored":38612992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39677952,"data_stored":39677952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39690240,"data_stored":39690240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:02.372 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:02.372 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11891", 'reported_seq': 11990, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.227810+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:06:59.227810+0000', 'last_peered': '2024-09-11T05:06:59.227810+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:06:59.227810+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5103616, 'num_objects': 623, 'num_object_clones': 0, 'num_object_copies': 2492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 623, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11891, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1668}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:02.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:02.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12152", 'reported_seq': 12272, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:01.595907+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:01.595907+0000', 'last_peered': '2024-09-11T05:07:01.595907+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:01.595907+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4797600, 'num_objects': 586, 'num_object_clones': 0, 'num_object_copies': 2344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12150, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1672}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:02.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:02.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11590", 'reported_seq': 11700, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.280896+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:06:59.280896+0000', 'last_peered': '2024-09-11T05:06:59.280896+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:06:59.280896+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4668008, 'num_objects': 571, 'num_object_clones': 0, 'num_object_copies': 2284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 571, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11589, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1640}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:02.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:02.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11846", 'reported_seq': 12447, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:01.716519+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:01.716519+0000', 'last_peered': '2024-09-11T05:07:01.716519+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:01.716519+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4666528, 'num_objects': 570, 'num_object_clones': 0, 'num_object_copies': 2280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11844, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 263, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:02.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:02.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11943", 'reported_seq': 12071, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.261148+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:06:59.261148+0000', 'last_peered': '2024-09-11T05:06:59.261148+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:06:59.261148+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4813984, 'num_objects': 588, 'num_object_clones': 0, 'num_object_copies': 2352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11941, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1633}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:02.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:02.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12025", 'reported_seq': 12136, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.292021+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:06:59.292021+0000', 'last_peered': '2024-09-11T05:06:59.292021+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:06:59.292021+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5038080, 'num_objects': 615, 'num_object_clones': 0, 'num_object_copies': 2460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12025, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1679}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:02.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:02.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12122", 'reported_seq': 12221, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:01.304789+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:01.304789+0000', 'last_peered': '2024-09-11T05:07:01.304789+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:01.304789+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4977824, 'num_objects': 608, 'num_object_clones': 0, 'num_object_copies': 2432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12120, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1699}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:02.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12160", 'reported_seq': 12285, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.421782+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:06:59.421782+0000', 'last_peered': '2024-09-11T05:06:59.421782+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:06:59.421782+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5138752, 'num_objects': 628, 'num_object_clones': 0, 'num_object_copies': 2512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12157, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1738}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11965", 'reported_seq': 12081, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.389884+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:06:59.389884+0000', 'last_peered': '2024-09-11T05:06:59.389884+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:06:59.389884+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4609184, 'num_objects': 563, 'num_object_clones': 0, 'num_object_copies': 2252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1743}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12015", 'reported_seq': 12137, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.290794+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:06:59.290794+0000', 'last_peered': '2024-09-11T05:06:59.290794+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:06:59.290794+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5222128, 'num_objects': 638, 'num_object_clones': 0, 'num_object_copies': 2552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12012, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1660}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11983", 'reported_seq': 12094, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.129429+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:06:59.129429+0000', 'last_peered': '2024-09-11T05:06:59.129429+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:06:59.129429+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4986016, 'num_objects': 609, 'num_object_clones': 0, 'num_object_copies': 2436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11981, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1655}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:02.375 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11859", 'reported_seq': 11983, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:06:59.329244+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:06:59.329244+0000', 'last_peered': '2024-09-11T05:06:59.329244+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:06:59.329244+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4502688, 'num_objects': 550, 'num_object_clones': 0, 'num_object_copies': 2200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11857, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1753}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:02.376 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:02.376 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12180", 'reported_seq': 12305, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:01.571185+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:01.571185+0000', 'last_peered': '2024-09-11T05:07:01.571185+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:01.571185+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4887712, 'num_objects': 597, 'num_object_clones': 0, 'num_object_copies': 2388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12178, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:02.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:03.210 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29447/38172 objects degraded (77.143%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":750,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9543,"data_bytes":78133144,"bytes_used":2824134656,"bytes_avail":383722921984,"bytes_total":386547056640,"degraded_objects":29447,"degraded_total":38172,"degraded_ratio":0.77142931992036035,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:03.237 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:03.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:03.916 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:03.938 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-11T05:07:01.909630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":78133144,"num_objects":9543,"num_object_clones":0,"num_object_copies":38172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164507,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757944,"kb_used_data":155040,"kb_used_omap":14,"kb_used_meta":2602801,"kb_avail":374729416,"statfs":{"total":386547056640,"available":383722921984,"internally_reserved":0,"allocated":158760960,"data_stored":157966924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2665268790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,11,7,20,23,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19374080,"num_objects":-2365,"num_object_clones":0,"num_object_copies":-9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005100"},"pg_stats":[{"pgid":"1.f","version":"37'11891","reported_seq":11990,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.227810+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:06:59.227810+0000","last_peered":"2024-09-11T05:06:59.227810+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:06:59.227810+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5103616,"num_objects":623,"num_object_clones":0,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12152","reported_seq":12272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.595907+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:01.595907+0000","last_peered":"2024-09-11T05:07:01.595907+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:01.595907+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4797600,"num_objects":586,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11590","reported_seq":11700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.280896+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:06:59.280896+0000","last_peered":"2024-09-11T05:06:59.280896+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:06:59.280896+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4668008,"num_objects":571,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11589,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11846","reported_seq":12447,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.716519+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:01.716519+0000","last_peered":"2024-09-11T05:07:01.716519+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:01.716519+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4666528,"num_objects":570,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11943","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.261148+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:06:59.261148+0000","last_peered":"2024-09-11T05:06:59.261148+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:06:59.261148+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4813984,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11941,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12025","reported_seq":12136,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.292021+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:06:59.292021+0000","last_peered":"2024-09-11T05:06:59.292021+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:06:59.292021+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5038080,"num_objects":615,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12025,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12122","reported_seq":12221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.304789+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:01.304789+0000","last_peered":"2024-09-11T05:07:01.304789+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:01.304789+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4977824,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12120,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1699}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12225","reported_seq":17087,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.772663+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:00.772663+0000","last_peered":"2024-09-11T05:07:00.772663+0000","last_clean":"2024-09-11T05:07:00.772663+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:00.772663+0000","last_undegraded":"2024-09-11T05:07:00.772663+0000","last_fullsized":"2024-09-11T05:07:00.772663+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4898816,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12160","reported_seq":12285,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.421782+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:06:59.421782+0000","last_peered":"2024-09-11T05:06:59.421782+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:06:59.421782+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5138752,"num_objects":628,"num_object_clones":0,"num_object_copies":2512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11965","reported_seq":12081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.389884+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:06:59.389884+0000","last_peered":"2024-09-11T05:06:59.389884+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:06:59.389884+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4609184,"num_objects":563,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12015","reported_seq":12137,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.290794+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:06:59.290794+0000","last_peered":"2024-09-11T05:06:59.290794+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:06:59.290794+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5222128,"num_objects":638,"num_object_clones":0,"num_object_copies":2552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1660}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11807","reported_seq":16413,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.601452+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:00.601452+0000","last_peered":"2024-09-11T05:07:00.601452+0000","last_clean":"2024-09-11T05:07:00.601452+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:00.601452+0000","last_undegraded":"2024-09-11T05:07:00.601452+0000","last_fullsized":"2024-09-11T05:07:00.601452+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4726784,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5617,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12100","reported_seq":16882,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.637744+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:00.637744+0000","last_peered":"2024-09-11T05:07:00.637744+0000","last_clean":"2024-09-11T05:07:00.637744+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:00.637744+0000","last_undegraded":"2024-09-11T05:07:00.637744+0000","last_fullsized":"2024-09-11T05:07:00.637744+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5095424,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11983","reported_seq":12094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.129429+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:06:59.129429+0000","last_peered":"2024-09-11T05:06:59.129429+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:06:59.129429+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986016,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1655}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11859","reported_seq":11983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:06:59.329244+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:06:59.329244+0000","last_peered":"2024-09-11T05:06:59.329244+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:06:59.329244+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4502688,"num_objects":550,"num_object_clones":0,"num_object_copies":2200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11857,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1753}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12180","reported_seq":12305,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.571185+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:01.571185+0000","last_peered":"2024-09-11T05:07:01.571185+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:01.571185+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4887712,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":78133144,"num_objects":9543,"num_object_clones":0,"num_object_copies":38172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164507,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155860992,"data_stored":155860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739092,"kb_used_data":39612,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93632748,"statfs":{"total":96636764160,"available":95879933952,"internally_reserved":0,"allocated":40562688,"data_stored":40364179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720284,"kb_used_data":39372,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93651556,"statfs":{"total":96636764160,"available":95899193344,"internally_reserved":0,"allocated":40316928,"data_stored":40118419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,9,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583848,"kb_used_data":38540,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93787992,"statfs":{"total":96636764160,"available":96038903808,"internally_reserved":0,"allocated":39464960,"data_stored":39266451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558362322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,6,9,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714720,"kb_used_data":37516,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93657120,"statfs":{"total":96636764160,"available":95904890880,"internally_reserved":0,"allocated":38416384,"data_stored":38217875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693432015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37879808,"data_stored":37879808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38612992,"data_stored":38612992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39677952,"data_stored":39677952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39690240,"data_stored":39690240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:06.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:07.594 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:07.594 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:07.620 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-11T05:07:05.911448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":73791384,"num_objects":9013,"num_object_clones":0,"num_object_copies":36052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165037,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160793,"ondisk_log_size":160793,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754600,"kb_used_data":147544,"kb_used_omap":14,"kb_used_meta":2606961,"kb_avail":374732760,"statfs":{"total":386547056640,"available":383726346240,"internally_reserved":0,"allocated":151085056,"data_stored":150291020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2669528630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,12,6,15,19,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20004864,"num_objects":-2442,"num_object_clones":0,"num_object_copies":-9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005246"},"pg_stats":[{"pgid":"1.f","version":"37'11947","reported_seq":12047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.281716+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:04.281716+0000","last_peered":"2024-09-11T05:07:04.281716+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:04.281716+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4644864,"num_objects":567,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1724}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12152","reported_seq":12272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.595907+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:01.595907+0000","last_peered":"2024-09-11T05:07:01.595907+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:01.595907+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4797600,"num_objects":586,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11651","reported_seq":11761,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.385058+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:04.385058+0000","last_peered":"2024-09-11T05:07:04.385058+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:04.385058+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4168296,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11650,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1701}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11846","reported_seq":12447,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.716519+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:01.716519+0000","last_peered":"2024-09-11T05:07:01.716519+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:01.716519+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4666528,"num_objects":570,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11995","reported_seq":12123,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.492259+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:04.492259+0000","last_peered":"2024-09-11T05:07:04.492259+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:04.492259+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4388000,"num_objects":536,"num_object_clones":0,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11993,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12087","reported_seq":12198,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.457807+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:04.457807+0000","last_peered":"2024-09-11T05:07:04.457807+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:04.457807+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530176,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1741}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12122","reported_seq":12221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.304789+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:01.304789+0000","last_peered":"2024-09-11T05:07:01.304789+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:01.304789+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4977824,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12120,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1699}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12225","reported_seq":17087,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.772663+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:00.772663+0000","last_peered":"2024-09-11T05:07:00.772663+0000","last_clean":"2024-09-11T05:07:00.772663+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:00.772663+0000","last_undegraded":"2024-09-11T05:07:00.772663+0000","last_fullsized":"2024-09-11T05:07:00.772663+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4898816,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12222","reported_seq":12350,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.419637+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:04.419637+0000","last_peered":"2024-09-11T05:07:04.419637+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:04.419637+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4630848,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12041","reported_seq":12158,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.380946+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:04.380946+0000","last_peered":"2024-09-11T05:07:04.380946+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:04.380946+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3986592,"num_objects":487,"num_object_clones":0,"num_object_copies":1948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1819}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12072","reported_seq":12194,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.170687+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:04.170687+0000","last_peered":"2024-09-11T05:07:04.170687+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:04.170687+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4755184,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11807","reported_seq":16413,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.601452+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:00.601452+0000","last_peered":"2024-09-11T05:07:00.601452+0000","last_clean":"2024-09-11T05:07:00.601452+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:00.601452+0000","last_undegraded":"2024-09-11T05:07:00.601452+0000","last_fullsized":"2024-09-11T05:07:00.601452+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4726784,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5617,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12100","reported_seq":16882,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:00.637744+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:00.637744+0000","last_peered":"2024-09-11T05:07:00.637744+0000","last_clean":"2024-09-11T05:07:00.637744+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:00.637744+0000","last_undegraded":"2024-09-11T05:07:00.637744+0000","last_fullsized":"2024-09-11T05:07:00.637744+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5095424,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12033","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.316855+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:04.316855+0000","last_peered":"2024-09-11T05:07:04.316855+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:04.316855+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576416,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1705}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11913","reported_seq":12037,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.250361+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:04.250361+0000","last_peered":"2024-09-11T05:07:04.250361+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:04.250361+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4060320,"num_objects":496,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11911,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12180","reported_seq":12305,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:01.571185+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:01.571185+0000","last_peered":"2024-09-11T05:07:01.571185+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:01.571185+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4887712,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":73791384,"num_objects":9013,"num_object_clones":0,"num_object_copies":36052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165037,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":148127744,"data_stored":148127744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160793,"ondisk_log_size":160793,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739496,"kb_used_data":35864,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93632344,"statfs":{"total":96636764160,"available":95879520256,"internally_reserved":0,"allocated":36724736,"data_stored":36526227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716536,"kb_used_data":35624,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93655304,"statfs":{"total":96636764160,"available":95903031296,"internally_reserved":0,"allocated":36478976,"data_stored":36280467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583848,"kb_used_data":38540,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93787992,"statfs":{"total":96636764160,"available":96038903808,"internally_reserved":0,"allocated":39464960,"data_stored":39266451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":558362322},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,6,9,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714720,"kb_used_data":37516,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93657120,"statfs":{"total":96636764160,"available":95904890880,"internally_reserved":0,"allocated":38416384,"data_stored":38217875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693432015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37879808,"data_stored":37879808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38612992,"data_stored":38612992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35831808,"data_stored":35831808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":35803136,"data_stored":35803136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:07.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:08.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:08.275 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:08.308 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-11T05:07:07.912360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":70137752,"num_objects":8567,"num_object_clones":0,"num_object_copies":34268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8567,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165483,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738764,"kb_used_data":140032,"kb_used_omap":14,"kb_used_meta":2598641,"kb_avail":374748596,"statfs":{"total":386547056640,"available":383742562304,"internally_reserved":0,"allocated":143392768,"data_stored":142598732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2661008950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,8,9,13,17,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17883136,"num_objects":-2183,"num_object_clones":0,"num_object_copies":-8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2183,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005358"},"pg_stats":[{"pgid":"1.f","version":"37'11947","reported_seq":12047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.281716+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:04.281716+0000","last_peered":"2024-09-11T05:07:04.281716+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:04.281716+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4644864,"num_objects":567,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1724}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12208","reported_seq":12327,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:06.786905+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:06.786905+0000","last_peered":"2024-09-11T05:07:06.786905+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:06.786905+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4338848,"num_objects":530,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12206,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1728}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11651","reported_seq":11761,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.385058+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:04.385058+0000","last_peered":"2024-09-11T05:07:04.385058+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:04.385058+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4168296,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11650,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1701}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11905","reported_seq":12598,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:06.709646+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:06.709646+0000","last_peered":"2024-09-11T05:07:06.709646+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:06.709646+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183200,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11903,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11995","reported_seq":12123,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.492259+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:04.492259+0000","last_peered":"2024-09-11T05:07:04.492259+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:04.492259+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4388000,"num_objects":536,"num_object_clones":0,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11993,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12087","reported_seq":12198,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.457807+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:04.457807+0000","last_peered":"2024-09-11T05:07:04.457807+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:04.457807+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530176,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1741}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12194","reported_seq":12293,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:06.493476+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:06.493476+0000","last_peered":"2024-09-11T05:07:06.493476+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:06.493476+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4388000,"num_objects":536,"num_object_clones":0,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12192,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12287","reported_seq":17147,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:05.854762+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:05.854762+0000","last_peered":"2024-09-11T05:07:05.854762+0000","last_clean":"2024-09-11T05:07:05.854762+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:05.854762+0000","last_undegraded":"2024-09-11T05:07:05.854762+0000","last_fullsized":"2024-09-11T05:07:05.854762+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":536,"num_object_clones":0,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12222","reported_seq":12350,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.419637+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:04.419637+0000","last_peered":"2024-09-11T05:07:04.419637+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:04.419637+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4630848,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12041","reported_seq":12158,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.380946+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:04.380946+0000","last_peered":"2024-09-11T05:07:04.380946+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:04.380946+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3986592,"num_objects":487,"num_object_clones":0,"num_object_copies":1948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1819}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12072","reported_seq":12194,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.170687+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:04.170687+0000","last_peered":"2024-09-11T05:07:04.170687+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:04.170687+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4755184,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11868","reported_seq":16474,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:05.830922+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:05.830922+0000","last_peered":"2024-09-11T05:07:05.830922+0000","last_clean":"2024-09-11T05:07:05.830922+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:05.830922+0000","last_undegraded":"2024-09-11T05:07:05.830922+0000","last_fullsized":"2024-09-11T05:07:05.830922+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4227072,"num_objects":516,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12166","reported_seq":16948,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:05.917927+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:05.917927+0000","last_peered":"2024-09-11T05:07:05.917927+0000","last_clean":"2024-09-11T05:07:05.917927+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:05.917927+0000","last_undegraded":"2024-09-11T05:07:05.917927+0000","last_fullsized":"2024-09-11T05:07:05.917927+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4554752,"num_objects":556,"num_object_clones":0,"num_object_copies":2224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12033","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.316855+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:04.316855+0000","last_peered":"2024-09-11T05:07:04.316855+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:04.316855+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576416,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1705}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11913","reported_seq":12037,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.250361+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:04.250361+0000","last_peered":"2024-09-11T05:07:04.250361+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:04.250361+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4060320,"num_objects":496,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11911,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12250","reported_seq":12373,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:06.713924+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:06.713924+0000","last_peered":"2024-09-11T05:07:06.713924+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:06.713924+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4314272,"num_objects":527,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12248,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":70137752,"num_objects":8567,"num_object_clones":0,"num_object_copies":34268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8567,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165483,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140263424,"data_stored":140263424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739496,"kb_used_data":35864,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93632344,"statfs":{"total":96636764160,"available":95879520256,"internally_reserved":0,"allocated":36724736,"data_stored":36526227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716536,"kb_used_data":35624,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93655304,"statfs":{"total":96636764160,"available":95903031296,"internally_reserved":0,"allocated":36478976,"data_stored":36280467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567676,"kb_used_data":34848,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93804164,"statfs":{"total":96636764160,"available":96055463936,"internally_reserved":0,"allocated":35684352,"data_stored":35485843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545582802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,4,7,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715056,"kb_used_data":33696,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93656784,"statfs":{"total":96636764160,"available":95904546816,"internally_reserved":0,"allocated":34504704,"data_stored":34306195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33914880,"data_stored":33914880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34713600,"data_stored":34713600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35831808,"data_stored":35831808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":35803136,"data_stored":35803136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:08.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:08.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11947", 'reported_seq': 12047, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.281716+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:04.281716+0000', 'last_peered': '2024-09-11T05:07:04.281716+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:04.281716+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4644864, 'num_objects': 567, 'num_object_clones': 0, 'num_object_copies': 2268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 567, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11947, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1724}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:08.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:08.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12208", 'reported_seq': 12327, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:06.786905+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:06.786905+0000', 'last_peered': '2024-09-11T05:07:06.786905+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:06.786905+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4338848, 'num_objects': 530, 'num_object_clones': 0, 'num_object_copies': 2120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12206, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1728}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:08.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:08.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11651", 'reported_seq': 11761, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.385058+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:04.385058+0000', 'last_peered': '2024-09-11T05:07:04.385058+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:04.385058+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4168296, 'num_objects': 510, 'num_object_clones': 0, 'num_object_copies': 2040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 510, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11650, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1701}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:08.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:08.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11905", 'reported_seq': 12598, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:06.709646+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:06.709646+0000', 'last_peered': '2024-09-11T05:07:06.709646+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:06.709646+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4183200, 'num_objects': 511, 'num_object_clones': 0, 'num_object_copies': 2044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1973, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11903, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 309, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:08.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:08.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11995", 'reported_seq': 12123, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.492259+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:04.492259+0000', 'last_peered': '2024-09-11T05:07:04.492259+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:04.492259+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4388000, 'num_objects': 536, 'num_object_clones': 0, 'num_object_copies': 2144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11993, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1685}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:08.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:08.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12087", 'reported_seq': 12198, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.457807+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:04.457807+0000', 'last_peered': '2024-09-11T05:07:04.457807+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:04.457807+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4530176, 'num_objects': 553, 'num_object_clones': 0, 'num_object_copies': 2212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12087, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1741}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:08.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:08.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12194", 'reported_seq': 12293, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:06.493476+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:06.493476+0000', 'last_peered': '2024-09-11T05:07:06.493476+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:06.493476+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4388000, 'num_objects': 536, 'num_object_clones': 0, 'num_object_copies': 2144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12192, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:08.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:08.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12222", 'reported_seq': 12350, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.419637+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:04.419637+0000', 'last_peered': '2024-09-11T05:07:04.419637+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:04.419637+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4630848, 'num_objects': 566, 'num_object_clones': 0, 'num_object_copies': 2264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12219, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1800}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:08.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:08.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12041", 'reported_seq': 12158, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.380946+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:04.380946+0000', 'last_peered': '2024-09-11T05:07:04.380946+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:04.380946+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3986592, 'num_objects': 487, 'num_object_clones': 0, 'num_object_copies': 1948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12039, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1819}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:08.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:08.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12072", 'reported_seq': 12194, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.170687+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:04.170687+0000', 'last_peered': '2024-09-11T05:07:04.170687+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:04.170687+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4755184, 'num_objects': 581, 'num_object_clones': 0, 'num_object_copies': 2324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12069, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1717}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:08.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:08.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12033", 'reported_seq': 12145, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.316855+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:04.316855+0000', 'last_peered': '2024-09-11T05:07:04.316855+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:04.316855+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4576416, 'num_objects': 559, 'num_object_clones': 0, 'num_object_copies': 2236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12031, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1705}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:08.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:08.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11913", 'reported_seq': 12037, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:04.250361+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:04.250361+0000', 'last_peered': '2024-09-11T05:07:04.250361+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:04.250361+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4060320, 'num_objects': 496, 'num_object_clones': 0, 'num_object_copies': 1984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11911, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1807}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:08.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:08.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12250", 'reported_seq': 12373, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:06.713924+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:06.713924+0000', 'last_peered': '2024-09-11T05:07:06.713924+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:06.713924+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4314272, 'num_objects': 527, 'num_object_clones': 0, 'num_object_copies': 2108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12248, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1761}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:08.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:09.073 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:09.073 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29401/34268 objects degraded (85.797%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":756,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8567,"data_bytes":70137752,"bytes_used":2804494336,"bytes_avail":383742562304,"bytes_total":386547056640,"degraded_objects":29401,"degraded_total":34268,"degraded_ratio":0.85797245243375742,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:09.099 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:09.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:09.768 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:09.795 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-11T05:07:07.912360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":70137752,"num_objects":8567,"num_object_clones":0,"num_object_copies":34268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8567,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165483,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738764,"kb_used_data":140032,"kb_used_omap":14,"kb_used_meta":2598641,"kb_avail":374748596,"statfs":{"total":386547056640,"available":383742562304,"internally_reserved":0,"allocated":143392768,"data_stored":142598732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2661008950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,8,9,13,17,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17883136,"num_objects":-2183,"num_object_clones":0,"num_object_copies":-8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2183,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005358"},"pg_stats":[{"pgid":"1.f","version":"37'11947","reported_seq":12047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.281716+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:04.281716+0000","last_peered":"2024-09-11T05:07:04.281716+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:04.281716+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4644864,"num_objects":567,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1724}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12208","reported_seq":12327,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:06.786905+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:06.786905+0000","last_peered":"2024-09-11T05:07:06.786905+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:06.786905+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4338848,"num_objects":530,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12206,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1728}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11651","reported_seq":11761,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.385058+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:04.385058+0000","last_peered":"2024-09-11T05:07:04.385058+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:04.385058+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4168296,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11650,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1701}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11905","reported_seq":12598,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:06.709646+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:06.709646+0000","last_peered":"2024-09-11T05:07:06.709646+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:06.709646+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183200,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11903,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11995","reported_seq":12123,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.492259+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:04.492259+0000","last_peered":"2024-09-11T05:07:04.492259+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:04.492259+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4388000,"num_objects":536,"num_object_clones":0,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11993,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12087","reported_seq":12198,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.457807+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:04.457807+0000","last_peered":"2024-09-11T05:07:04.457807+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:04.457807+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530176,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1741}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12194","reported_seq":12293,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:06.493476+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:06.493476+0000","last_peered":"2024-09-11T05:07:06.493476+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:06.493476+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4388000,"num_objects":536,"num_object_clones":0,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12192,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12287","reported_seq":17147,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:05.854762+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:05.854762+0000","last_peered":"2024-09-11T05:07:05.854762+0000","last_clean":"2024-09-11T05:07:05.854762+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:05.854762+0000","last_undegraded":"2024-09-11T05:07:05.854762+0000","last_fullsized":"2024-09-11T05:07:05.854762+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":536,"num_object_clones":0,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12222","reported_seq":12350,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.419637+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:04.419637+0000","last_peered":"2024-09-11T05:07:04.419637+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:04.419637+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4630848,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12041","reported_seq":12158,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.380946+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:04.380946+0000","last_peered":"2024-09-11T05:07:04.380946+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:04.380946+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3986592,"num_objects":487,"num_object_clones":0,"num_object_copies":1948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1819}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12072","reported_seq":12194,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.170687+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:04.170687+0000","last_peered":"2024-09-11T05:07:04.170687+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:04.170687+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4755184,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11868","reported_seq":16474,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:05.830922+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:05.830922+0000","last_peered":"2024-09-11T05:07:05.830922+0000","last_clean":"2024-09-11T05:07:05.830922+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:05.830922+0000","last_undegraded":"2024-09-11T05:07:05.830922+0000","last_fullsized":"2024-09-11T05:07:05.830922+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4227072,"num_objects":516,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12166","reported_seq":16948,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:05.917927+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:05.917927+0000","last_peered":"2024-09-11T05:07:05.917927+0000","last_clean":"2024-09-11T05:07:05.917927+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:05.917927+0000","last_undegraded":"2024-09-11T05:07:05.917927+0000","last_fullsized":"2024-09-11T05:07:05.917927+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4554752,"num_objects":556,"num_object_clones":0,"num_object_copies":2224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12033","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.316855+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:04.316855+0000","last_peered":"2024-09-11T05:07:04.316855+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:04.316855+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576416,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1705}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11913","reported_seq":12037,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:04.250361+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:04.250361+0000","last_peered":"2024-09-11T05:07:04.250361+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:04.250361+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4060320,"num_objects":496,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11911,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12250","reported_seq":12373,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:06.713924+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:06.713924+0000","last_peered":"2024-09-11T05:07:06.713924+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:06.713924+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4314272,"num_objects":527,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12248,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":70137752,"num_objects":8567,"num_object_clones":0,"num_object_copies":34268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8567,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165483,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140263424,"data_stored":140263424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739496,"kb_used_data":35864,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93632344,"statfs":{"total":96636764160,"available":95879520256,"internally_reserved":0,"allocated":36724736,"data_stored":36526227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716536,"kb_used_data":35624,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93655304,"statfs":{"total":96636764160,"available":95903031296,"internally_reserved":0,"allocated":36478976,"data_stored":36280467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567676,"kb_used_data":34848,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93804164,"statfs":{"total":96636764160,"available":96055463936,"internally_reserved":0,"allocated":35684352,"data_stored":35485843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":545582802},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,4,7,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715056,"kb_used_data":33696,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93656784,"statfs":{"total":96636764160,"available":95904546816,"internally_reserved":0,"allocated":34504704,"data_stored":34306195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33914880,"data_stored":33914880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34713600,"data_stored":34713600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35831808,"data_stored":35831808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":35803136,"data_stored":35803136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:12.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:13.465 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:13.466 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:13.494 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-11T05:07:11.914250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62388120,"num_objects":7621,"num_object_clones":0,"num_object_copies":30484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166429,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740336,"kb_used_data":124952,"kb_used_omap":14,"kb_used_meta":2615281,"kb_avail":374747024,"statfs":{"total":386547056640,"available":383740952576,"internally_reserved":0,"allocated":127950848,"data_stored":127156812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2678048310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,7,11,11,15,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19193856,"num_objects":-2343,"num_object_clones":0,"num_object_copies":-9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005549"},"pg_stats":[{"pgid":"1.f","version":"37'12013","reported_seq":12112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.587441+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:09.587441+0000","last_peered":"2024-09-11T05:07:09.587441+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:09.587441+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4104192,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12013,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12273","reported_seq":12393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.758251+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:11.758251+0000","last_peered":"2024-09-11T05:07:11.758251+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:11.758251+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3806368,"num_objects":465,"num_object_clones":0,"num_object_copies":1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12271,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11714","reported_seq":11824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.454462+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:09.454462+0000","last_peered":"2024-09-11T05:07:09.454462+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:09.454462+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3652200,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11713,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1764}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11955","reported_seq":12745,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.771432+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:11.771432+0000","last_peered":"2024-09-11T05:07:11.771432+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:11.771432+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3773600,"num_objects":461,"num_object_clones":0,"num_object_copies":1844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11953,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12074","reported_seq":12201,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.553565+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:09.553565+0000","last_peered":"2024-09-11T05:07:09.553565+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:09.553565+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3740832,"num_objects":457,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12148","reported_seq":12259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.555783+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:09.555783+0000","last_peered":"2024-09-11T05:07:09.555783+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:09.555783+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4030464,"num_objects":492,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12148,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12267","reported_seq":12366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.811152+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:11.811152+0000","last_peered":"2024-09-11T05:07:11.811152+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:11.811152+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3789984,"num_objects":463,"num_object_clones":0,"num_object_copies":1852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1844}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12357","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:11.064865+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:11.064865+0000","last_peered":"2024-09-11T05:07:11.064865+0000","last_clean":"2024-09-11T05:07:11.064865+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:11.064865+0000","last_undegraded":"2024-09-11T05:07:11.064865+0000","last_fullsized":"2024-09-11T05:07:11.064865+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3817472,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3067,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12276","reported_seq":12404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.632017+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:09.632017+0000","last_peered":"2024-09-11T05:07:09.632017+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:09.632017+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4188480,"num_objects":512,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12273,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1854}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12086","reported_seq":12203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.348813+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:09.348813+0000","last_peered":"2024-09-11T05:07:09.348813+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:09.348813+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3617952,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1864}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12129","reported_seq":12251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.647494+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:09.647494+0000","last_peered":"2024-09-11T05:07:09.647494+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:09.647494+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4288240,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11921","reported_seq":16527,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:10.959932+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:10.959932+0000","last_peered":"2024-09-11T05:07:10.959932+0000","last_clean":"2024-09-11T05:07:10.959932+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:10.959932+0000","last_undegraded":"2024-09-11T05:07:10.959932+0000","last_fullsized":"2024-09-11T05:07:10.959932+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3792896,"num_objects":463,"num_object_clones":0,"num_object_copies":1852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12204","reported_seq":16985,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:11.051948+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:11.051948+0000","last_peered":"2024-09-11T05:07:11.051948+0000","last_clean":"2024-09-11T05:07:11.051948+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:11.051948+0000","last_undegraded":"2024-09-11T05:07:11.051948+0000","last_fullsized":"2024-09-11T05:07:11.051948+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4243456,"num_objects":518,"num_object_clones":0,"num_object_copies":2072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12100","reported_seq":12212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.545797+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:09.545797+0000","last_peered":"2024-09-11T05:07:09.545797+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:09.545797+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4027552,"num_objects":492,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1772}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11957","reported_seq":12081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.299139+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:09.299139+0000","last_peered":"2024-09-11T05:07:09.299139+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:09.299139+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3699872,"num_objects":452,"num_object_clones":0,"num_object_copies":1808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11955,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12311","reported_seq":12436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.740302+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:11.740302+0000","last_peered":"2024-09-11T05:07:11.740302+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:11.740302+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814560,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12309,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":62388120,"num_objects":7621,"num_object_clones":0,"num_object_copies":30484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166429,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124551168,"data_stored":124551168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739836,"kb_used_data":32040,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93632004,"statfs":{"total":96636764160,"available":95879172096,"internally_reserved":0,"allocated":32808960,"data_stored":32610451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,4,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716992,"kb_used_data":31904,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93654848,"statfs":{"total":96636764160,"available":95902564352,"internally_reserved":0,"allocated":32669696,"data_stored":32471187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568032,"kb_used_data":31056,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93803808,"statfs":{"total":96636764160,"available":96055099392,"internally_reserved":0,"allocated":31801344,"data_stored":31602835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549842642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,4,4,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715476,"kb_used_data":29952,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93656364,"statfs":{"total":96636764160,"available":95904116736,"internally_reserved":0,"allocated":30670848,"data_stored":30472339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30027776,"data_stored":30027776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30756864,"data_stored":30756864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31940608,"data_stored":31940608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31825920,"data_stored":31825920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:13.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:14.121 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:14.144 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-11T05:07:13.914973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62388120,"num_objects":7621,"num_object_clones":0,"num_object_copies":30484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166429,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740336,"kb_used_data":124952,"kb_used_omap":14,"kb_used_meta":2615281,"kb_avail":374747024,"statfs":{"total":386547056640,"available":383740952576,"internally_reserved":0,"allocated":127950848,"data_stored":127156812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2678048310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,7,11,11,15,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15745024,"num_objects":-1922,"num_object_clones":0,"num_object_copies":-7688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1922,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005342"},"pg_stats":[{"pgid":"1.f","version":"37'12013","reported_seq":12112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.587441+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:09.587441+0000","last_peered":"2024-09-11T05:07:09.587441+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:09.587441+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4104192,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12013,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12273","reported_seq":12393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.758251+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:11.758251+0000","last_peered":"2024-09-11T05:07:11.758251+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:11.758251+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3806368,"num_objects":465,"num_object_clones":0,"num_object_copies":1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12271,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11714","reported_seq":11824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.454462+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:09.454462+0000","last_peered":"2024-09-11T05:07:09.454462+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:09.454462+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3652200,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11713,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1764}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11955","reported_seq":12745,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.771432+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:11.771432+0000","last_peered":"2024-09-11T05:07:11.771432+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:11.771432+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3773600,"num_objects":461,"num_object_clones":0,"num_object_copies":1844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11953,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12074","reported_seq":12201,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.553565+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:09.553565+0000","last_peered":"2024-09-11T05:07:09.553565+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:09.553565+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3740832,"num_objects":457,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12148","reported_seq":12259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.555783+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:09.555783+0000","last_peered":"2024-09-11T05:07:09.555783+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:09.555783+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4030464,"num_objects":492,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12148,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12267","reported_seq":12366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.811152+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:11.811152+0000","last_peered":"2024-09-11T05:07:11.811152+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:11.811152+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3789984,"num_objects":463,"num_object_clones":0,"num_object_copies":1852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1844}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12357","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:11.064865+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:11.064865+0000","last_peered":"2024-09-11T05:07:11.064865+0000","last_clean":"2024-09-11T05:07:11.064865+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:11.064865+0000","last_undegraded":"2024-09-11T05:07:11.064865+0000","last_fullsized":"2024-09-11T05:07:11.064865+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3817472,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3067,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12276","reported_seq":12404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.632017+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:09.632017+0000","last_peered":"2024-09-11T05:07:09.632017+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:09.632017+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4188480,"num_objects":512,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12273,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1854}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12086","reported_seq":12203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.348813+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:09.348813+0000","last_peered":"2024-09-11T05:07:09.348813+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:09.348813+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3617952,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1864}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12129","reported_seq":12251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.647494+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:09.647494+0000","last_peered":"2024-09-11T05:07:09.647494+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:09.647494+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4288240,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11921","reported_seq":16527,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:10.959932+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:10.959932+0000","last_peered":"2024-09-11T05:07:10.959932+0000","last_clean":"2024-09-11T05:07:10.959932+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:10.959932+0000","last_undegraded":"2024-09-11T05:07:10.959932+0000","last_fullsized":"2024-09-11T05:07:10.959932+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3792896,"num_objects":463,"num_object_clones":0,"num_object_copies":1852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12204","reported_seq":16985,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:11.051948+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:11.051948+0000","last_peered":"2024-09-11T05:07:11.051948+0000","last_clean":"2024-09-11T05:07:11.051948+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:11.051948+0000","last_undegraded":"2024-09-11T05:07:11.051948+0000","last_fullsized":"2024-09-11T05:07:11.051948+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4243456,"num_objects":518,"num_object_clones":0,"num_object_copies":2072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12100","reported_seq":12212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.545797+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:09.545797+0000","last_peered":"2024-09-11T05:07:09.545797+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:09.545797+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4027552,"num_objects":492,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1772}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11957","reported_seq":12081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.299139+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:09.299139+0000","last_peered":"2024-09-11T05:07:09.299139+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:09.299139+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3699872,"num_objects":452,"num_object_clones":0,"num_object_copies":1808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11955,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12311","reported_seq":12436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.740302+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:11.740302+0000","last_peered":"2024-09-11T05:07:11.740302+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:11.740302+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814560,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12309,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":62388120,"num_objects":7621,"num_object_clones":0,"num_object_copies":30484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166429,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124551168,"data_stored":124551168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739836,"kb_used_data":32040,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93632004,"statfs":{"total":96636764160,"available":95879172096,"internally_reserved":0,"allocated":32808960,"data_stored":32610451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,4,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716992,"kb_used_data":31904,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93654848,"statfs":{"total":96636764160,"available":95902564352,"internally_reserved":0,"allocated":32669696,"data_stored":32471187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568032,"kb_used_data":31056,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93803808,"statfs":{"total":96636764160,"available":96055099392,"internally_reserved":0,"allocated":31801344,"data_stored":31602835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549842642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,4,4,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715476,"kb_used_data":29952,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93656364,"statfs":{"total":96636764160,"available":95904116736,"internally_reserved":0,"allocated":30670848,"data_stored":30472339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30027776,"data_stored":30027776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30756864,"data_stored":30756864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31940608,"data_stored":31940608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31825920,"data_stored":31825920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:14.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:14.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12013", 'reported_seq': 12112, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.587441+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:09.587441+0000', 'last_peered': '2024-09-11T05:07:09.587441+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:09.587441+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4104192, 'num_objects': 501, 'num_object_clones': 0, 'num_object_copies': 2004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12013, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1790}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12273", 'reported_seq': 12393, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:11.758251+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:11.758251+0000', 'last_peered': '2024-09-11T05:07:11.758251+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:11.758251+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3806368, 'num_objects': 465, 'num_object_clones': 0, 'num_object_copies': 1860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12271, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1793}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11714", 'reported_seq': 11824, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.454462+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:09.454462+0000', 'last_peered': '2024-09-11T05:07:09.454462+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:09.454462+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3652200, 'num_objects': 447, 'num_object_clones': 0, 'num_object_copies': 1788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 447, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11713, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1764}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11955", 'reported_seq': 12745, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:11.771432+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:11.771432+0000', 'last_peered': '2024-09-11T05:07:11.771432+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:11.771432+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3773600, 'num_objects': 461, 'num_object_clones': 0, 'num_object_copies': 1844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1925, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11953, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 357, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12074", 'reported_seq': 12201, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.553565+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:09.553565+0000', 'last_peered': '2024-09-11T05:07:09.553565+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:09.553565+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3740832, 'num_objects': 457, 'num_object_clones': 0, 'num_object_copies': 1828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12072, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1764}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12148", 'reported_seq': 12259, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.555783+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:09.555783+0000', 'last_peered': '2024-09-11T05:07:09.555783+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:09.555783+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4030464, 'num_objects': 492, 'num_object_clones': 0, 'num_object_copies': 1968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12148, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1802}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:14.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12267", 'reported_seq': 12366, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:11.811152+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:11.811152+0000', 'last_peered': '2024-09-11T05:07:11.811152+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:11.811152+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3789984, 'num_objects': 463, 'num_object_clones': 0, 'num_object_copies': 1852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12265, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1844}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12276", 'reported_seq': 12404, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.632017+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:09.632017+0000', 'last_peered': '2024-09-11T05:07:09.632017+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:09.632017+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4188480, 'num_objects': 512, 'num_object_clones': 0, 'num_object_copies': 2048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12273, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1854}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12086", 'reported_seq': 12203, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.348813+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:09.348813+0000', 'last_peered': '2024-09-11T05:07:09.348813+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:09.348813+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3617952, 'num_objects': 442, 'num_object_clones': 0, 'num_object_copies': 1768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12084, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1864}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12129", 'reported_seq': 12251, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.647494+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:09.647494+0000', 'last_peered': '2024-09-11T05:07:09.647494+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:09.647494+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4288240, 'num_objects': 524, 'num_object_clones': 0, 'num_object_copies': 2096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12126, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1774}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12100", 'reported_seq': 12212, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.545797+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:09.545797+0000', 'last_peered': '2024-09-11T05:07:09.545797+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:09.545797+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4027552, 'num_objects': 492, 'num_object_clones': 0, 'num_object_copies': 1968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1772}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:14.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11957", 'reported_seq': 12081, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:09.299139+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:09.299139+0000', 'last_peered': '2024-09-11T05:07:09.299139+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:09.299139+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3699872, 'num_objects': 452, 'num_object_clones': 0, 'num_object_copies': 1808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11955, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:14.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:14.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12311", 'reported_seq': 12436, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:11.740302+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:11.740302+0000', 'last_peered': '2024-09-11T05:07:11.740302+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:11.740302+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3814560, 'num_objects': 466, 'num_object_clones': 0, 'num_object_copies': 1864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12309, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:14.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:14.964 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:14.964 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29353/30484 objects degraded (96.290%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":761,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7621,"data_bytes":62388120,"bytes_used":2806104064,"bytes_avail":383740952576,"bytes_total":386547056640,"degraded_objects":29353,"degraded_total":30484,"degraded_ratio":0.96289856974150378,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":160},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:14.994 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:14.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:15.624 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:15.625 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:15.651 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-11T05:07:13.914973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62388120,"num_objects":7621,"num_object_clones":0,"num_object_copies":30484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166429,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740336,"kb_used_data":124952,"kb_used_omap":14,"kb_used_meta":2615281,"kb_avail":374747024,"statfs":{"total":386547056640,"available":383740952576,"internally_reserved":0,"allocated":127950848,"data_stored":127156812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2678048310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,7,11,11,15,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15745024,"num_objects":-1922,"num_object_clones":0,"num_object_copies":-7688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1922,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005342"},"pg_stats":[{"pgid":"1.f","version":"37'12013","reported_seq":12112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.587441+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:09.587441+0000","last_peered":"2024-09-11T05:07:09.587441+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:09.587441+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4104192,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12013,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12273","reported_seq":12393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.758251+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:11.758251+0000","last_peered":"2024-09-11T05:07:11.758251+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:11.758251+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3806368,"num_objects":465,"num_object_clones":0,"num_object_copies":1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12271,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11714","reported_seq":11824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.454462+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:09.454462+0000","last_peered":"2024-09-11T05:07:09.454462+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:09.454462+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3652200,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11713,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1764}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11955","reported_seq":12745,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.771432+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:11.771432+0000","last_peered":"2024-09-11T05:07:11.771432+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:11.771432+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3773600,"num_objects":461,"num_object_clones":0,"num_object_copies":1844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11953,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12074","reported_seq":12201,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.553565+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:09.553565+0000","last_peered":"2024-09-11T05:07:09.553565+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:09.553565+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3740832,"num_objects":457,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12148","reported_seq":12259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.555783+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:09.555783+0000","last_peered":"2024-09-11T05:07:09.555783+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:09.555783+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4030464,"num_objects":492,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12148,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12267","reported_seq":12366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.811152+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:11.811152+0000","last_peered":"2024-09-11T05:07:11.811152+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:11.811152+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3789984,"num_objects":463,"num_object_clones":0,"num_object_copies":1852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1844}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12357","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:11.064865+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:11.064865+0000","last_peered":"2024-09-11T05:07:11.064865+0000","last_clean":"2024-09-11T05:07:11.064865+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:11.064865+0000","last_undegraded":"2024-09-11T05:07:11.064865+0000","last_fullsized":"2024-09-11T05:07:11.064865+0000","mapping_epoch":34,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3817472,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3067,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12276","reported_seq":12404,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.632017+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:09.632017+0000","last_peered":"2024-09-11T05:07:09.632017+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:09.632017+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4188480,"num_objects":512,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12273,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1854}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12086","reported_seq":12203,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.348813+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:09.348813+0000","last_peered":"2024-09-11T05:07:09.348813+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:09.348813+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3617952,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1864}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12129","reported_seq":12251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.647494+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:09.647494+0000","last_peered":"2024-09-11T05:07:09.647494+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:09.647494+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4288240,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11921","reported_seq":16527,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:10.959932+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:10.959932+0000","last_peered":"2024-09-11T05:07:10.959932+0000","last_clean":"2024-09-11T05:07:10.959932+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:10.959932+0000","last_undegraded":"2024-09-11T05:07:10.959932+0000","last_fullsized":"2024-09-11T05:07:10.959932+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3792896,"num_objects":463,"num_object_clones":0,"num_object_copies":1852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12204","reported_seq":16985,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:11.051948+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:11.051948+0000","last_peered":"2024-09-11T05:07:11.051948+0000","last_clean":"2024-09-11T05:07:11.051948+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:11.051948+0000","last_undegraded":"2024-09-11T05:07:11.051948+0000","last_fullsized":"2024-09-11T05:07:11.051948+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4243456,"num_objects":518,"num_object_clones":0,"num_object_copies":2072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12100","reported_seq":12212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.545797+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:09.545797+0000","last_peered":"2024-09-11T05:07:09.545797+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:09.545797+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4027552,"num_objects":492,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1772}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11957","reported_seq":12081,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:09.299139+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:09.299139+0000","last_peered":"2024-09-11T05:07:09.299139+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:09.299139+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3699872,"num_objects":452,"num_object_clones":0,"num_object_copies":1808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11955,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12311","reported_seq":12436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:11.740302+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:11.740302+0000","last_peered":"2024-09-11T05:07:11.740302+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:11.740302+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814560,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12309,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":62388120,"num_objects":7621,"num_object_clones":0,"num_object_copies":30484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166429,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124551168,"data_stored":124551168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739836,"kb_used_data":32040,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93632004,"statfs":{"total":96636764160,"available":95879172096,"internally_reserved":0,"allocated":32808960,"data_stored":32610451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,4,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716992,"kb_used_data":31904,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93654848,"statfs":{"total":96636764160,"available":95902564352,"internally_reserved":0,"allocated":32669696,"data_stored":32471187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568032,"kb_used_data":31056,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93803808,"statfs":{"total":96636764160,"available":96055099392,"internally_reserved":0,"allocated":31801344,"data_stored":31602835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":549842642},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,4,4,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715476,"kb_used_data":29952,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93656364,"statfs":{"total":96636764160,"available":95904116736,"internally_reserved":0,"allocated":30670848,"data_stored":30472339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30027776,"data_stored":30027776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30756864,"data_stored":30756864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31940608,"data_stored":31940608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31825920,"data_stored":31825920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:18.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:19.348 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:19.348 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:19.370 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-11T05:07:17.916168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54622104,"num_objects":6673,"num_object_clones":0,"num_object_copies":26692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167377,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":670,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742376,"kb_used_data":108968,"kb_used_omap":14,"kb_used_meta":2633329,"kb_avail":374744984,"statfs":{"total":386547056640,"available":383738863616,"internally_reserved":0,"allocated":111583232,"data_stored":110789196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2696529462},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,6,8,12,13,17,16,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19169280,"num_objects":-2340,"num_object_clones":0,"num_object_copies":-9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004719"},"pg_stats":[{"pgid":"1.f","version":"37'12090","reported_seq":12190,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.589818+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:14.589818+0000","last_peered":"2024-09-11T05:07:14.589818+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:14.589818+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":424,"num_object_clones":0,"num_object_copies":1696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12090,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12328","reported_seq":12448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.948591+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:16.948591+0000","last_peered":"2024-09-11T05:07:16.948591+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:16.948591+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3355808,"num_objects":410,"num_object_clones":0,"num_object_copies":1640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11770","reported_seq":11880,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.418700+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:14.418700+0000","last_peered":"2024-09-11T05:07:14.418700+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:14.418700+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193448,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11769,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1820}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12014","reported_seq":12901,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.988546+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:16.988546+0000","last_peered":"2024-09-11T05:07:16.988546+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:16.988546+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3290272,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":406,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12136","reported_seq":12264,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.589840+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:14.589840+0000","last_peered":"2024-09-11T05:07:14.589840+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:14.589840+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3232928,"num_objects":395,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12134,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12209","reported_seq":12320,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.646639+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:14.646639+0000","last_peered":"2024-09-11T05:07:14.646639+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:14.646639+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3530752,"num_objects":431,"num_object_clones":0,"num_object_copies":1724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12209,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12327","reported_seq":12422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.811365+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:16.811365+0000","last_peered":"2024-09-11T05:07:16.811365+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:16.811365+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3298464,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12325,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1904}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12410","reported_seq":17273,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.083256+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:16.083256+0000","last_peered":"2024-09-11T05:07:16.083256+0000","last_clean":"2024-09-11T05:07:16.083256+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:16.083256+0000","last_undegraded":"2024-09-11T05:07:16.083256+0000","last_fullsized":"2024-09-11T05:07:16.083256+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3383296,"num_objects":413,"num_object_clones":0,"num_object_copies":1652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12346","reported_seq":12474,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.528117+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:14.528117+0000","last_peered":"2024-09-11T05:07:14.528117+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:14.528117+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3615040,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12343,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1924}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12132","reported_seq":12249,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.515407+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:14.515407+0000","last_peered":"2024-09-11T05:07:14.515407+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:14.515407+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3241120,"num_objects":396,"num_object_clones":0,"num_object_copies":1584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12130,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12184","reported_seq":12306,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.527949+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:14.527949+0000","last_peered":"2024-09-11T05:07:14.527949+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:14.527949+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3837680,"num_objects":469,"num_object_clones":0,"num_object_copies":1876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12181,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1829}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11986","reported_seq":16592,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.147609+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:16.147609+0000","last_peered":"2024-09-11T05:07:16.147609+0000","last_clean":"2024-09-11T05:07:16.147609+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:16.147609+0000","last_undegraded":"2024-09-11T05:07:16.147609+0000","last_fullsized":"2024-09-11T05:07:16.147609+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260416,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5796,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12275","reported_seq":17057,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.032341+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:16.032341+0000","last_peered":"2024-09-11T05:07:16.032341+0000","last_clean":"2024-09-11T05:07:16.032341+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:16.032341+0000","last_undegraded":"2024-09-11T05:07:16.032341+0000","last_fullsized":"2024-09-11T05:07:16.032341+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3661824,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":422,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12151","reported_seq":12263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.439810+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:14.439810+0000","last_peered":"2024-09-11T05:07:14.439810+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:14.439810+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3609760,"num_objects":441,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12149,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1823}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12006","reported_seq":12129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:14.463554+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:14.463554+0000","last_peered":"2024-09-11T05:07:14.463554+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:14.463554+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3298464,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1900}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12369","reported_seq":12494,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.674792+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:16.674792+0000","last_peered":"2024-09-11T05:07:16.674792+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:16.674792+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3339424,"num_objects":408,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12367,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54622104,"num_objects":6673,"num_object_clones":0,"num_object_copies":26692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167377,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":670,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108687360,"data_stored":108687360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723472,"kb_used_data":28160,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93648368,"statfs":{"total":96636764160,"available":95895928832,"internally_reserved":0,"allocated":28835840,"data_stored":28637331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":711978704},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,0,6,1,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717308,"kb_used_data":28076,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93654532,"statfs":{"total":96636764160,"available":95902240768,"internally_reserved":0,"allocated":28749824,"data_stored":28551315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585716,"kb_used_data":26532,"kb_used_omap":4,"kb_used_meta":559163,"kb_avail":93786124,"statfs":{"total":96636764160,"available":96036990976,"internally_reserved":0,"allocated":27168768,"data_stored":26970259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572583634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,3,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715880,"kb_used_data":26200,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93655960,"statfs":{"total":96636764160,"available":95903703040,"internally_reserved":0,"allocated":26828800,"data_stored":26630291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706211535},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,6,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26066944,"data_stored":26066944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26763264,"data_stored":26763264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":27967488,"data_stored":27967488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27889664,"data_stored":27889664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:19.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:19.975 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:19.975 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:20.003 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-11T05:07:19.917122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":50116504,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167927,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":670,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725724,"kb_used_data":100612,"kb_used_omap":14,"kb_used_meta":2625009,"kb_avail":374761636,"statfs":{"total":386547056640,"available":383755915264,"internally_reserved":0,"allocated":103026688,"data_stored":102232652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2688009782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,11,3,14,11,16,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20021248,"num_objects":-2444,"num_object_clones":0,"num_object_copies":-9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004761"},"pg_stats":[{"pgid":"1.f","version":"37'12156","reported_seq":12256,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.755483+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:19.755483+0000","last_peered":"2024-09-11T05:07:19.755483+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:19.755483+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932736,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12156,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12328","reported_seq":12448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.948591+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:16.948591+0000","last_peered":"2024-09-11T05:07:16.948591+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:16.948591+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3355808,"num_objects":410,"num_object_clones":0,"num_object_copies":1640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11824","reported_seq":11934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.352853+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:19.352853+0000","last_peered":"2024-09-11T05:07:19.352853+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:19.352853+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2751080,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11823,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12014","reported_seq":12901,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.988546+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:16.988546+0000","last_peered":"2024-09-11T05:07:16.988546+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:16.988546+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3290272,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":406,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12185","reported_seq":12313,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.586065+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:19.586065+0000","last_peered":"2024-09-11T05:07:19.586065+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:19.586065+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831520,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12282","reported_seq":12389,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.825555+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:19.825555+0000","last_peered":"2024-09-11T05:07:19.825555+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:19.825555+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932736,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12327","reported_seq":12422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.811365+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:16.811365+0000","last_peered":"2024-09-11T05:07:16.811365+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:16.811365+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3298464,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12325,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1904}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12410","reported_seq":17273,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.083256+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:16.083256+0000","last_peered":"2024-09-11T05:07:16.083256+0000","last_clean":"2024-09-11T05:07:16.083256+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:16.083256+0000","last_undegraded":"2024-09-11T05:07:16.083256+0000","last_fullsized":"2024-09-11T05:07:16.083256+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3383296,"num_objects":413,"num_object_clones":0,"num_object_copies":1652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12407","reported_seq":12535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.409509+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:19.409509+0000","last_peered":"2024-09-11T05:07:19.409509+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:19.409509+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"28'2400","ondisk_log_start":"28'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3115328,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12404,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12199","reported_seq":12316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.456900+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:19.456900+0000","last_peered":"2024-09-11T05:07:19.456900+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:19.456900+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2692256,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12197,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12244","reported_seq":12366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.679478+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:19.679478+0000","last_peered":"2024-09-11T05:07:19.679478+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:19.679478+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3346160,"num_objects":409,"num_object_clones":0,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12241,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11986","reported_seq":16592,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.147609+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:16.147609+0000","last_peered":"2024-09-11T05:07:16.147609+0000","last_clean":"2024-09-11T05:07:16.147609+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:16.147609+0000","last_undegraded":"2024-09-11T05:07:16.147609+0000","last_fullsized":"2024-09-11T05:07:16.147609+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260416,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5796,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12275","reported_seq":17057,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.032341+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:16.032341+0000","last_peered":"2024-09-11T05:07:16.032341+0000","last_clean":"2024-09-11T05:07:16.032341+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:16.032341+0000","last_undegraded":"2024-09-11T05:07:16.032341+0000","last_fullsized":"2024-09-11T05:07:16.032341+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3661824,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":422,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12219","reported_seq":12331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.700042+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:19.700042+0000","last_peered":"2024-09-11T05:07:19.700042+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:19.700042+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3052704,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1891}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12058","reported_seq":12182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.570402+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:19.570402+0000","last_peered":"2024-09-11T05:07:19.570402+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:19.570402+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2872480,"num_objects":351,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12056,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12369","reported_seq":12494,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.674792+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:16.674792+0000","last_peered":"2024-09-11T05:07:16.674792+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:16.674792+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3339424,"num_objects":408,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12367,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":50116504,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167927,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":670,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":100700160,"data_stored":100700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723208,"kb_used_data":23720,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93648632,"statfs":{"total":96636764160,"available":95896199168,"internally_reserved":0,"allocated":24289280,"data_stored":24090771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700920,"kb_used_data":24160,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93670920,"statfs":{"total":96636764160,"available":95919022080,"internally_reserved":0,"allocated":24739840,"data_stored":24541331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585716,"kb_used_data":26532,"kb_used_omap":4,"kb_used_meta":559163,"kb_avail":93786124,"statfs":{"total":96636764160,"available":96036990976,"internally_reserved":0,"allocated":27168768,"data_stored":26970259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572583634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,3,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715880,"kb_used_data":26200,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93655960,"statfs":{"total":96636764160,"available":95903703040,"internally_reserved":0,"allocated":26828800,"data_stored":26630291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706211535},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,6,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26066944,"data_stored":26066944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26763264,"data_stored":26763264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24035328,"data_stored":24035328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23834624,"data_stored":23834624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:20.004 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:20.004 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12156", 'reported_seq': 12256, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.755483+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:19.755483+0000', 'last_peered': '2024-09-11T05:07:19.755483+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:19.755483+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2932736, 'num_objects': 358, 'num_object_clones': 0, 'num_object_copies': 1432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12156, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1933}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12328", 'reported_seq': 12448, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:16.948591+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:16.948591+0000', 'last_peered': '2024-09-11T05:07:16.948591+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:16.948591+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3355808, 'num_objects': 410, 'num_object_clones': 0, 'num_object_copies': 1640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12326, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1848}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11824", 'reported_seq': 11934, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.352853+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:19.352853+0000', 'last_peered': '2024-09-11T05:07:19.352853+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:19.352853+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2751080, 'num_objects': 337, 'num_object_clones': 0, 'num_object_copies': 1348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 337, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11823, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1874}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12014", 'reported_seq': 12901, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:16.988546+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:16.988546+0000', 'last_peered': '2024-09-11T05:07:16.988546+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:16.988546+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3290272, 'num_objects': 402, 'num_object_clones': 0, 'num_object_copies': 1608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1876, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12012, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 406, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1474}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12185", 'reported_seq': 12313, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.586065+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:19.586065+0000', 'last_peered': '2024-09-11T05:07:19.586065+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:19.586065+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2831520, 'num_objects': 346, 'num_object_clones': 0, 'num_object_copies': 1384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12183, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1875}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12282", 'reported_seq': 12389, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.825555+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:19.825555+0000', 'last_peered': '2024-09-11T05:07:19.825555+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:19.825555+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2932736, 'num_objects': 358, 'num_object_clones': 0, 'num_object_copies': 1432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12282, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1936}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:20.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12327", 'reported_seq': 12422, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:16.811365+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:16.811365+0000', 'last_peered': '2024-09-11T05:07:16.811365+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:16.811365+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3298464, 'num_objects': 403, 'num_object_clones': 0, 'num_object_copies': 1612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12325, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1904}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12407", 'reported_seq': 12535, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.409509+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:19.409509+0000', 'last_peered': '2024-09-11T05:07:19.409509+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:19.409509+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3115328, 'num_objects': 381, 'num_object_clones': 0, 'num_object_copies': 1524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12404, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12199", 'reported_seq': 12316, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.456900+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:19.456900+0000', 'last_peered': '2024-09-11T05:07:19.456900+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:19.456900+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2692256, 'num_objects': 329, 'num_object_clones': 0, 'num_object_copies': 1316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12197, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12244", 'reported_seq': 12366, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.679478+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:19.679478+0000', 'last_peered': '2024-09-11T05:07:19.679478+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:19.679478+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3346160, 'num_objects': 409, 'num_object_clones': 0, 'num_object_copies': 1636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12241, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1889}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12219", 'reported_seq': 12331, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.700042+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:19.700042+0000', 'last_peered': '2024-09-11T05:07:19.700042+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:19.700042+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3052704, 'num_objects': 373, 'num_object_clones': 0, 'num_object_copies': 1492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12217, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1891}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:20.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12058", 'reported_seq': 12182, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.570402+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:19.570402+0000', 'last_peered': '2024-09-11T05:07:19.570402+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:19.570402+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2872480, 'num_objects': 351, 'num_object_clones': 0, 'num_object_copies': 1404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12056, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:20.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:20.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12369", 'reported_seq': 12494, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:16.674792+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:16.674792+0000', 'last_peered': '2024-09-11T05:07:16.674792+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:16.674792+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3339424, 'num_objects': 408, 'num_object_clones': 0, 'num_object_copies': 1632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12367, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:20.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:20.773 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29304/24492 objects degraded (119.647%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":767,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6123,"data_bytes":50116504,"bytes_used":2791141376,"bytes_avail":383755915264,"bytes_total":386547056640,"degraded_objects":29304,"degraded_total":24492,"degraded_ratio":1.1964723174914258,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":203},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:20.799 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:20.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:21.446 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-11T05:07:19.917122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":50116504,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167927,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":670,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725724,"kb_used_data":100612,"kb_used_omap":14,"kb_used_meta":2625009,"kb_avail":374761636,"statfs":{"total":386547056640,"available":383755915264,"internally_reserved":0,"allocated":103026688,"data_stored":102232652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2688009782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,11,3,14,11,16,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20021248,"num_objects":-2444,"num_object_clones":0,"num_object_copies":-9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004761"},"pg_stats":[{"pgid":"1.f","version":"37'12156","reported_seq":12256,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.755483+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:19.755483+0000","last_peered":"2024-09-11T05:07:19.755483+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:19.755483+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932736,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12156,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12328","reported_seq":12448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.948591+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:16.948591+0000","last_peered":"2024-09-11T05:07:16.948591+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:16.948591+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3355808,"num_objects":410,"num_object_clones":0,"num_object_copies":1640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11824","reported_seq":11934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.352853+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:19.352853+0000","last_peered":"2024-09-11T05:07:19.352853+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:19.352853+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2751080,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11823,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12014","reported_seq":12901,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.988546+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:16.988546+0000","last_peered":"2024-09-11T05:07:16.988546+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:16.988546+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3290272,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":406,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12185","reported_seq":12313,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.586065+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:19.586065+0000","last_peered":"2024-09-11T05:07:19.586065+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:19.586065+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831520,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12282","reported_seq":12389,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.825555+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:19.825555+0000","last_peered":"2024-09-11T05:07:19.825555+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:19.825555+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932736,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12327","reported_seq":12422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.811365+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:16.811365+0000","last_peered":"2024-09-11T05:07:16.811365+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:16.811365+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3298464,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12325,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1904}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12410","reported_seq":17273,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.083256+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:16.083256+0000","last_peered":"2024-09-11T05:07:16.083256+0000","last_clean":"2024-09-11T05:07:16.083256+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:16.083256+0000","last_undegraded":"2024-09-11T05:07:16.083256+0000","last_fullsized":"2024-09-11T05:07:16.083256+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3383296,"num_objects":413,"num_object_clones":0,"num_object_copies":1652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12407","reported_seq":12535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.409509+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:19.409509+0000","last_peered":"2024-09-11T05:07:19.409509+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:19.409509+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"28'2400","ondisk_log_start":"28'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3115328,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12404,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12199","reported_seq":12316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.456900+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:19.456900+0000","last_peered":"2024-09-11T05:07:19.456900+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:19.456900+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2692256,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12197,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12244","reported_seq":12366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.679478+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:19.679478+0000","last_peered":"2024-09-11T05:07:19.679478+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:19.679478+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3346160,"num_objects":409,"num_object_clones":0,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12241,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11986","reported_seq":16592,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.147609+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:16.147609+0000","last_peered":"2024-09-11T05:07:16.147609+0000","last_clean":"2024-09-11T05:07:16.147609+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:16.147609+0000","last_undegraded":"2024-09-11T05:07:16.147609+0000","last_fullsized":"2024-09-11T05:07:16.147609+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260416,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5796,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12275","reported_seq":17057,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:16.032341+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:16.032341+0000","last_peered":"2024-09-11T05:07:16.032341+0000","last_clean":"2024-09-11T05:07:16.032341+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:16.032341+0000","last_undegraded":"2024-09-11T05:07:16.032341+0000","last_fullsized":"2024-09-11T05:07:16.032341+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3661824,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":422,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12219","reported_seq":12331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.700042+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:19.700042+0000","last_peered":"2024-09-11T05:07:19.700042+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:19.700042+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3052704,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1891}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12058","reported_seq":12182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.570402+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:19.570402+0000","last_peered":"2024-09-11T05:07:19.570402+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:19.570402+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2872480,"num_objects":351,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12056,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12369","reported_seq":12494,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:16.674792+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:16.674792+0000","last_peered":"2024-09-11T05:07:16.674792+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:16.674792+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3339424,"num_objects":408,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12367,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":50116504,"num_objects":6123,"num_object_clones":0,"num_object_copies":24492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167927,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":670,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":100700160,"data_stored":100700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723208,"kb_used_data":23720,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93648632,"statfs":{"total":96636764160,"available":95896199168,"internally_reserved":0,"allocated":24289280,"data_stored":24090771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700920,"kb_used_data":24160,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93670920,"statfs":{"total":96636764160,"available":95919022080,"internally_reserved":0,"allocated":24739840,"data_stored":24541331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585716,"kb_used_data":26532,"kb_used_omap":4,"kb_used_meta":559163,"kb_avail":93786124,"statfs":{"total":96636764160,"available":96036990976,"internally_reserved":0,"allocated":27168768,"data_stored":26970259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":572583634},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,3,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715880,"kb_used_data":26200,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93655960,"statfs":{"total":96636764160,"available":95903703040,"internally_reserved":0,"allocated":26828800,"data_stored":26630291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706211535},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,6,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26066944,"data_stored":26066944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26763264,"data_stored":26763264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24035328,"data_stored":24035328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23834624,"data_stored":23834624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:24.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:25.124 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:25.142 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-11T05:07:23.921900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46323608,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168390,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160946,"ondisk_log_size":160946,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709728,"kb_used_data":92932,"kb_used_omap":14,"kb_used_meta":2616689,"kb_avail":374777632,"statfs":{"total":386547056640,"available":383772295168,"internally_reserved":0,"allocated":95162368,"data_stored":94368332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2679490102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,6,3,16,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16064512,"num_objects":-1961,"num_object_clones":0,"num_object_copies":-7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007650"},"pg_stats":[{"pgid":"1.f","version":"37'12156","reported_seq":12256,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.755483+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:19.755483+0000","last_peered":"2024-09-11T05:07:19.755483+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:19.755483+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932736,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12156,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12391","reported_seq":12510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.848021+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:21.848021+0000","last_peered":"2024-09-11T05:07:21.848021+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:21.848021+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2839712,"num_objects":347,"num_object_clones":0,"num_object_copies":1388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12389,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11824","reported_seq":11934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.352853+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:19.352853+0000","last_peered":"2024-09-11T05:07:19.352853+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:19.352853+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2751080,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11823,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12078","reported_seq":13042,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.952979+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:21.952979+0000","last_peered":"2024-09-11T05:07:21.952979+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:21.952979+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2765984,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12076,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12185","reported_seq":12313,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.586065+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:19.586065+0000","last_peered":"2024-09-11T05:07:19.586065+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:19.586065+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831520,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12282","reported_seq":12389,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.825555+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:19.825555+0000","last_peered":"2024-09-11T05:07:19.825555+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:19.825555+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932736,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12393","reported_seq":12490,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.886969+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:21.886969+0000","last_peered":"2024-09-11T05:07:21.886969+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:21.886969+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2757792,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12391,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1970}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12483","reported_seq":17346,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.361561+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:21.361561+0000","last_peered":"2024-09-11T05:07:21.361561+0000","last_clean":"2024-09-11T05:07:21.361561+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:21.361561+0000","last_undegraded":"2024-09-11T05:07:21.361561+0000","last_fullsized":"2024-09-11T05:07:21.361561+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2785280,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12407","reported_seq":12535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.409509+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:19.409509+0000","last_peered":"2024-09-11T05:07:19.409509+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:19.409509+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"28'2400","ondisk_log_start":"28'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3115328,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12404,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12199","reported_seq":12316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.456900+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:19.456900+0000","last_peered":"2024-09-11T05:07:19.456900+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:19.456900+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2692256,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12197,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12244","reported_seq":12366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.679478+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:19.679478+0000","last_peered":"2024-09-11T05:07:19.679478+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:19.679478+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3346160,"num_objects":409,"num_object_clones":0,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12241,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12047","reported_seq":16653,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.255379+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:21.255379+0000","last_peered":"2024-09-11T05:07:21.255379+0000","last_clean":"2024-09-11T05:07:21.255379+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:21.255379+0000","last_undegraded":"2024-09-11T05:07:21.255379+0000","last_fullsized":"2024-09-11T05:07:21.255379+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2760704,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12349","reported_seq":17130,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.371423+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:21.371423+0000","last_peered":"2024-09-11T05:07:21.371423+0000","last_clean":"2024-09-11T05:07:21.371423+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:21.371423+0000","last_undegraded":"2024-09-11T05:07:21.371423+0000","last_fullsized":"2024-09-11T05:07:21.371423+0000","mapping_epoch":36,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3055616,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12219","reported_seq":12331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.700042+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:19.700042+0000","last_peered":"2024-09-11T05:07:19.700042+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:19.700042+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3052704,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1891}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12058","reported_seq":12182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.570402+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:19.570402+0000","last_peered":"2024-09-11T05:07:19.570402+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:19.570402+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2872480,"num_objects":351,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12056,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12431","reported_seq":12556,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.997087+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:21.997087+0000","last_peered":"2024-09-11T05:07:21.997087+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:21.997087+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831520,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12429,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":46323608,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168390,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":92827648,"data_stored":92827648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160946,"ondisk_log_size":160946,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723208,"kb_used_data":23720,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93648632,"statfs":{"total":96636764160,"available":95896199168,"internally_reserved":0,"allocated":24289280,"data_stored":24090771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700920,"kb_used_data":24160,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93670920,"statfs":{"total":96636764160,"available":95919022080,"internally_reserved":0,"allocated":24739840,"data_stored":24541331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585964,"kb_used_data":22616,"kb_used_omap":4,"kb_used_meta":563323,"kb_avail":93785876,"statfs":{"total":96636764160,"available":96036737024,"internally_reserved":0,"allocated":23158784,"data_stored":22960275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576843474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699636,"kb_used_data":22436,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93672204,"statfs":{"total":96636764160,"available":95920336896,"internally_reserved":0,"allocated":22974464,"data_stored":22775955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693432015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22167552,"data_stored":22167552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22790144,"data_stored":22790144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24035328,"data_stored":24035328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23834624,"data_stored":23834624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:25.144 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:25.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:25.826 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:25.845 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-11T05:07:23.921900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46323608,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168390,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160946,"ondisk_log_size":160946,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709728,"kb_used_data":92932,"kb_used_omap":14,"kb_used_meta":2616689,"kb_avail":374777632,"statfs":{"total":386547056640,"available":383772295168,"internally_reserved":0,"allocated":95162368,"data_stored":94368332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2679490102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,6,3,16,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16064512,"num_objects":-1961,"num_object_clones":0,"num_object_copies":-7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1961,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007650"},"pg_stats":[{"pgid":"1.f","version":"37'12156","reported_seq":12256,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.755483+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:19.755483+0000","last_peered":"2024-09-11T05:07:19.755483+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:19.755483+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932736,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12156,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12391","reported_seq":12510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.848021+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:21.848021+0000","last_peered":"2024-09-11T05:07:21.848021+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:21.848021+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2839712,"num_objects":347,"num_object_clones":0,"num_object_copies":1388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12389,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11824","reported_seq":11934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.352853+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:19.352853+0000","last_peered":"2024-09-11T05:07:19.352853+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:19.352853+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2751080,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11823,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12078","reported_seq":13042,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.952979+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:21.952979+0000","last_peered":"2024-09-11T05:07:21.952979+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:21.952979+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2765984,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12076,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12185","reported_seq":12313,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.586065+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:19.586065+0000","last_peered":"2024-09-11T05:07:19.586065+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:19.586065+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831520,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12282","reported_seq":12389,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.825555+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:19.825555+0000","last_peered":"2024-09-11T05:07:19.825555+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:19.825555+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932736,"num_objects":358,"num_object_clones":0,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12393","reported_seq":12490,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.886969+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:21.886969+0000","last_peered":"2024-09-11T05:07:21.886969+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:21.886969+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2757792,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12391,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1970}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12483","reported_seq":17346,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.361561+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:21.361561+0000","last_peered":"2024-09-11T05:07:21.361561+0000","last_clean":"2024-09-11T05:07:21.361561+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:21.361561+0000","last_undegraded":"2024-09-11T05:07:21.361561+0000","last_fullsized":"2024-09-11T05:07:21.361561+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2785280,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12407","reported_seq":12535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.409509+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:19.409509+0000","last_peered":"2024-09-11T05:07:19.409509+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:19.409509+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"28'2400","ondisk_log_start":"28'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3115328,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12404,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12199","reported_seq":12316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.456900+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:19.456900+0000","last_peered":"2024-09-11T05:07:19.456900+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:19.456900+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2692256,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12197,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12244","reported_seq":12366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.679478+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:19.679478+0000","last_peered":"2024-09-11T05:07:19.679478+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:19.679478+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3346160,"num_objects":409,"num_object_clones":0,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12241,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12047","reported_seq":16653,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.255379+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:21.255379+0000","last_peered":"2024-09-11T05:07:21.255379+0000","last_clean":"2024-09-11T05:07:21.255379+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:21.255379+0000","last_undegraded":"2024-09-11T05:07:21.255379+0000","last_fullsized":"2024-09-11T05:07:21.255379+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2760704,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12349","reported_seq":17130,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.371423+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:21.371423+0000","last_peered":"2024-09-11T05:07:21.371423+0000","last_clean":"2024-09-11T05:07:21.371423+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:21.371423+0000","last_undegraded":"2024-09-11T05:07:21.371423+0000","last_fullsized":"2024-09-11T05:07:21.371423+0000","mapping_epoch":36,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3055616,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12219","reported_seq":12331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.700042+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:19.700042+0000","last_peered":"2024-09-11T05:07:19.700042+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:19.700042+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3052704,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1891}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12058","reported_seq":12182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:19.570402+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:19.570402+0000","last_peered":"2024-09-11T05:07:19.570402+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:19.570402+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2872480,"num_objects":351,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12056,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12431","reported_seq":12556,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.997087+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:21.997087+0000","last_peered":"2024-09-11T05:07:21.997087+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:21.997087+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831520,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12429,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":46323608,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168390,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":92827648,"data_stored":92827648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160946,"ondisk_log_size":160946,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723208,"kb_used_data":23720,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93648632,"statfs":{"total":96636764160,"available":95896199168,"internally_reserved":0,"allocated":24289280,"data_stored":24090771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":716238544},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700920,"kb_used_data":24160,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93670920,"statfs":{"total":96636764160,"available":95919022080,"internally_reserved":0,"allocated":24739840,"data_stored":24541331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585964,"kb_used_data":22616,"kb_used_omap":4,"kb_used_meta":563323,"kb_avail":93785876,"statfs":{"total":96636764160,"available":96036737024,"internally_reserved":0,"allocated":23158784,"data_stored":22960275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576843474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699636,"kb_used_data":22436,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93672204,"statfs":{"total":96636764160,"available":95920336896,"internally_reserved":0,"allocated":22974464,"data_stored":22775955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693432015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22167552,"data_stored":22167552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22790144,"data_stored":22790144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24035328,"data_stored":24035328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23834624,"data_stored":23834624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:25.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:25.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12156", 'reported_seq': 12256, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.755483+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:19.755483+0000', 'last_peered': '2024-09-11T05:07:19.755483+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:19.755483+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2932736, 'num_objects': 358, 'num_object_clones': 0, 'num_object_copies': 1432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12156, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1933}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:25.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:25.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12391", 'reported_seq': 12510, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:21.848021+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:21.848021+0000', 'last_peered': '2024-09-11T05:07:21.848021+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:21.848021+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2839712, 'num_objects': 347, 'num_object_clones': 0, 'num_object_copies': 1388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 347, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12389, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:25.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:25.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11824", 'reported_seq': 11934, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.352853+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:19.352853+0000', 'last_peered': '2024-09-11T05:07:19.352853+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:19.352853+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2751080, 'num_objects': 337, 'num_object_clones': 0, 'num_object_copies': 1348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 337, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11823, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1874}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:25.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:25.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12078", 'reported_seq': 13042, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:21.952979+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:21.952979+0000', 'last_peered': '2024-09-11T05:07:21.952979+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:21.952979+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2765984, 'num_objects': 338, 'num_object_clones': 0, 'num_object_copies': 1352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1838, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12076, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 444, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1500}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:25.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:25.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12185", 'reported_seq': 12313, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.586065+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:19.586065+0000', 'last_peered': '2024-09-11T05:07:19.586065+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:19.586065+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2831520, 'num_objects': 346, 'num_object_clones': 0, 'num_object_copies': 1384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12183, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1875}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:25.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:25.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12282", 'reported_seq': 12389, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.825555+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:19.825555+0000', 'last_peered': '2024-09-11T05:07:19.825555+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:19.825555+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2932736, 'num_objects': 358, 'num_object_clones': 0, 'num_object_copies': 1432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12282, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1936}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:25.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:25.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12393", 'reported_seq': 12490, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:21.886969+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:21.886969+0000', 'last_peered': '2024-09-11T05:07:21.886969+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:21.886969+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2757792, 'num_objects': 337, 'num_object_clones': 0, 'num_object_copies': 1348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12391, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1970}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:25.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:25.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12407", 'reported_seq': 12535, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.409509+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:19.409509+0000', 'last_peered': '2024-09-11T05:07:19.409509+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:19.409509+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3115328, 'num_objects': 381, 'num_object_clones': 0, 'num_object_copies': 1524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12404, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:25.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:25.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12199", 'reported_seq': 12316, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.456900+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:19.456900+0000', 'last_peered': '2024-09-11T05:07:19.456900+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:19.456900+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2692256, 'num_objects': 329, 'num_object_clones': 0, 'num_object_copies': 1316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12197, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:25.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:25.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12244", 'reported_seq': 12366, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.679478+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:19.679478+0000', 'last_peered': '2024-09-11T05:07:19.679478+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:19.679478+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3346160, 'num_objects': 409, 'num_object_clones': 0, 'num_object_copies': 1636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12241, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1889}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:25.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:25.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12219", 'reported_seq': 12331, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.700042+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:19.700042+0000', 'last_peered': '2024-09-11T05:07:19.700042+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:19.700042+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3052704, 'num_objects': 373, 'num_object_clones': 0, 'num_object_copies': 1492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12217, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1891}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:25.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:25.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12058", 'reported_seq': 12182, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:19.570402+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:19.570402+0000', 'last_peered': '2024-09-11T05:07:19.570402+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:19.570402+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2872480, 'num_objects': 351, 'num_object_clones': 0, 'num_object_copies': 1404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12056, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:25.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:25.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12431", 'reported_seq': 12556, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:21.997087+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:21.997087+0000', 'last_peered': '2024-09-11T05:07:21.997087+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:21.997087+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2831520, 'num_objects': 346, 'num_object_clones': 0, 'num_object_copies': 1384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12429, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1942}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:25.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:26.616 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:26.616 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29266/20512 objects degraded (142.677%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":773,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5128,"data_bytes":41965464,"bytes_used":2775293952,"bytes_avail":383771762688,"bytes_total":386547056640,"degraded_objects":29266,"degraded_total":20512,"degraded_ratio":1.4267745709828392,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:26.644 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:26.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:27.335 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-11T05:07:25.922855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41965464,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168922,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2710248,"kb_used_data":85156,"kb_used_omap":14,"kb_used_meta":2625009,"kb_avail":374777112,"statfs":{"total":386547056640,"available":383771762688,"internally_reserved":0,"allocated":87199744,"data_stored":86405708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2688009782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,12,5,4,22,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20422656,"num_objects":-2493,"num_object_clones":0,"num_object_copies":-9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007881"},"pg_stats":[{"pgid":"1.f","version":"37'12208","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.987180+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:24.987180+0000","last_peered":"2024-09-11T05:07:24.987180+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:24.987180+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2506752,"num_objects":306,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12208,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12391","reported_seq":12510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.848021+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:21.848021+0000","last_peered":"2024-09-11T05:07:21.848021+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:21.848021+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2839712,"num_objects":347,"num_object_clones":0,"num_object_copies":1388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12389,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11872","reported_seq":11982,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.218541+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:24.218541+0000","last_peered":"2024-09-11T05:07:24.218541+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:24.218541+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2357864,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11871,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1922}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12078","reported_seq":13042,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.952979+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:21.952979+0000","last_peered":"2024-09-11T05:07:21.952979+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:21.952979+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2765984,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12076,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12239","reported_seq":12367,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.867700+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:24.867700+0000","last_peered":"2024-09-11T05:07:24.867700+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:24.867700+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389152,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12338","reported_seq":12448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.888535+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:24.888535+0000","last_peered":"2024-09-11T05:07:24.888535+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:24.888535+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2473984,"num_objects":302,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12393","reported_seq":12490,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.886969+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:21.886969+0000","last_peered":"2024-09-11T05:07:21.886969+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:21.886969+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2757792,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12391,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1970}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12483","reported_seq":17346,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.361561+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:21.361561+0000","last_peered":"2024-09-11T05:07:21.361561+0000","last_clean":"2024-09-11T05:07:21.361561+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:21.361561+0000","last_undegraded":"2024-09-11T05:07:21.361561+0000","last_fullsized":"2024-09-11T05:07:21.361561+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2785280,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12479","reported_seq":12607,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.960738+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:24.960738+0000","last_peered":"2024-09-11T05:07:24.960738+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:24.960738+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"28'2400","ondisk_log_start":"28'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2525504,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12476,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12246","reported_seq":12362,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.672976+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:24.672976+0000","last_peered":"2024-09-11T05:07:24.672976+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:24.672976+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2307232,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12244,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2024}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12331","reported_seq":12452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.928925+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:24.928925+0000","last_peered":"2024-09-11T05:07:24.928925+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:24.928925+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2633456,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12047","reported_seq":16653,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.255379+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:21.255379+0000","last_peered":"2024-09-11T05:07:21.255379+0000","last_clean":"2024-09-11T05:07:21.255379+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:21.255379+0000","last_undegraded":"2024-09-11T05:07:21.255379+0000","last_fullsized":"2024-09-11T05:07:21.255379+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2760704,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12349","reported_seq":17130,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:21.371423+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:21.371423+0000","last_peered":"2024-09-11T05:07:21.371423+0000","last_clean":"2024-09-11T05:07:21.371423+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:21.371423+0000","last_undegraded":"2024-09-11T05:07:21.371423+0000","last_fullsized":"2024-09-11T05:07:21.371423+0000","mapping_epoch":36,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3055616,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12268","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.888092+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:24.888092+0000","last_peered":"2024-09-11T05:07:24.888092+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:24.888092+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2651296,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12266,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12125","reported_seq":12249,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.478239+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:24.478239+0000","last_peered":"2024-09-11T05:07:24.478239+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:24.478239+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2323616,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12431","reported_seq":12556,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:21.997087+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:21.997087+0000","last_peered":"2024-09-11T05:07:21.997087+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:21.997087+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2831520,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12429,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":41965464,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168922,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":84705280,"data_stored":84705280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723340,"kb_used_data":19708,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93648500,"statfs":{"total":96636764160,"available":95896064000,"internally_reserved":0,"allocated":20180992,"data_stored":19982483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701308,"kb_used_data":20396,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93670532,"statfs":{"total":96636764160,"available":95918624768,"internally_reserved":0,"allocated":20885504,"data_stored":20686995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,0,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585964,"kb_used_data":22616,"kb_used_omap":4,"kb_used_meta":563323,"kb_avail":93785876,"statfs":{"total":96636764160,"available":96036737024,"internally_reserved":0,"allocated":23158784,"data_stored":22960275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":576843474},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699636,"kb_used_data":22436,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93672204,"statfs":{"total":96636764160,"available":95920336896,"internally_reserved":0,"allocated":22974464,"data_stored":22775955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693432015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22167552,"data_stored":22167552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22790144,"data_stored":22790144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19984384,"data_stored":19984384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19763200,"data_stored":19763200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:30.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:30.991 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:31.017 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-11T05:07:29.924218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36960152,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169533,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712444,"kb_used_data":73468,"kb_used_omap":14,"kb_used_meta":2638897,"kb_avail":374774916,"statfs":{"total":386547056640,"available":383769513984,"internally_reserved":0,"allocated":75231232,"data_stored":74437196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2702231094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,9,15,9,8,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17661952,"num_objects":-2156,"num_object_clones":0,"num_object_copies":-8624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008049"},"pg_stats":[{"pgid":"1.f","version":"37'12208","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.987180+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:24.987180+0000","last_peered":"2024-09-11T05:07:24.987180+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:24.987180+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2506752,"num_objects":306,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12208,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12465","reported_seq":12585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.946844+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:26.946844+0000","last_peered":"2024-09-11T05:07:26.946844+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:26.946844+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2233504,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12463,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11927","reported_seq":12036,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.768273+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:29.768273+0000","last_peered":"2024-09-11T05:07:29.768273+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:29.768273+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1907304,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11926,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12132","reported_seq":13204,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:27.045649+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:27.045649+0000","last_peered":"2024-09-11T05:07:27.045649+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:27.045649+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2323616,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12130,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12239","reported_seq":12367,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.867700+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:24.867700+0000","last_peered":"2024-09-11T05:07:24.867700+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:24.867700+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389152,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12338","reported_seq":12448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.888535+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:24.888535+0000","last_peered":"2024-09-11T05:07:24.888535+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:24.888535+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2473984,"num_objects":302,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12463","reported_seq":12562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.875593+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:26.875593+0000","last_peered":"2024-09-11T05:07:26.875593+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:26.875593+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2184352,"num_objects":267,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2040}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12536","reported_seq":17399,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:26.360670+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:26.360670+0000","last_peered":"2024-09-11T05:07:26.360670+0000","last_clean":"2024-09-11T05:07:26.360670+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:26.360670+0000","last_undegraded":"2024-09-11T05:07:26.360670+0000","last_fullsized":"2024-09-11T05:07:26.360670+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2351104,"num_objects":287,"num_object_clones":0,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12479","reported_seq":12607,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.960738+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:24.960738+0000","last_peered":"2024-09-11T05:07:24.960738+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:24.960738+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"28'2400","ondisk_log_start":"28'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2525504,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12476,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12312","reported_seq":12429,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.722952+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:29.722952+0000","last_peered":"2024-09-11T05:07:29.722952+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:29.722952+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1766560,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12310,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12331","reported_seq":12452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.928925+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:24.928925+0000","last_peered":"2024-09-11T05:07:24.928925+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:24.928925+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2633456,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12100","reported_seq":16706,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:26.390230+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:26.390230+0000","last_peered":"2024-09-11T05:07:26.390230+0000","last_clean":"2024-09-11T05:07:26.390230+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:26.390230+0000","last_undegraded":"2024-09-11T05:07:26.390230+0000","last_fullsized":"2024-09-11T05:07:26.390230+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2326528,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12420","reported_seq":17202,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:26.258792+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:26.258792+0000","last_peered":"2024-09-11T05:07:26.258792+0000","last_clean":"2024-09-11T05:07:26.258792+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:26.258792+0000","last_undegraded":"2024-09-11T05:07:26.258792+0000","last_fullsized":"2024-09-11T05:07:26.258792+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2473984,"num_objects":302,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12268","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.888092+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:24.888092+0000","last_peered":"2024-09-11T05:07:24.888092+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:24.888092+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2651296,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12266,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12183","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.605442+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:29.605442+0000","last_peered":"2024-09-11T05:07:29.605442+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:29.605442+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1848480,"num_objects":226,"num_object_clones":0,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12181,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2077}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12488","reported_seq":12612,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.805286+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:26.805286+0000","last_peered":"2024-09-11T05:07:26.805286+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:26.805286+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2364576,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12486,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":36960152,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169533,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":72634368,"data_stored":72634368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723340,"kb_used_data":19708,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93648500,"statfs":{"total":96636764160,"available":95896064000,"internally_reserved":0,"allocated":20180992,"data_stored":19982483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701664,"kb_used_data":16592,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93670176,"statfs":{"total":96636764160,"available":95918260224,"internally_reserved":0,"allocated":16990208,"data_stored":16791699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587556,"kb_used_data":18644,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93784284,"statfs":{"total":96636764160,"available":96035106816,"internally_reserved":0,"allocated":19091456,"data_stored":18892947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582545106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699884,"kb_used_data":18524,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93671956,"statfs":{"total":96636764160,"available":95920082944,"internally_reserved":0,"allocated":18968576,"data_stored":18770067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,12,3,2,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18235392,"data_stored":18235392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18690048,"data_stored":18690048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15945728,"data_stored":15945728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19763200,"data_stored":19763200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:31.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:31.684 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:31.685 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:31.715 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-11T05:07:29.924218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36960152,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169533,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712444,"kb_used_data":73468,"kb_used_omap":14,"kb_used_meta":2638897,"kb_avail":374774916,"statfs":{"total":386547056640,"available":383769513984,"internally_reserved":0,"allocated":75231232,"data_stored":74437196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2702231094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,9,15,9,8,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17661952,"num_objects":-2156,"num_object_clones":0,"num_object_copies":-8624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008049"},"pg_stats":[{"pgid":"1.f","version":"37'12208","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.987180+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:24.987180+0000","last_peered":"2024-09-11T05:07:24.987180+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:24.987180+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2506752,"num_objects":306,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12208,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12465","reported_seq":12585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.946844+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:26.946844+0000","last_peered":"2024-09-11T05:07:26.946844+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:26.946844+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2233504,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12463,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11927","reported_seq":12036,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.768273+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:29.768273+0000","last_peered":"2024-09-11T05:07:29.768273+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:29.768273+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1907304,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11926,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12132","reported_seq":13204,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:27.045649+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:27.045649+0000","last_peered":"2024-09-11T05:07:27.045649+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:27.045649+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2323616,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12130,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12239","reported_seq":12367,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.867700+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:24.867700+0000","last_peered":"2024-09-11T05:07:24.867700+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:24.867700+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389152,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12338","reported_seq":12448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.888535+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:24.888535+0000","last_peered":"2024-09-11T05:07:24.888535+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:24.888535+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2473984,"num_objects":302,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12463","reported_seq":12562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.875593+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:26.875593+0000","last_peered":"2024-09-11T05:07:26.875593+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:26.875593+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2184352,"num_objects":267,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2040}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12536","reported_seq":17399,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:26.360670+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:26.360670+0000","last_peered":"2024-09-11T05:07:26.360670+0000","last_clean":"2024-09-11T05:07:26.360670+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:26.360670+0000","last_undegraded":"2024-09-11T05:07:26.360670+0000","last_fullsized":"2024-09-11T05:07:26.360670+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2351104,"num_objects":287,"num_object_clones":0,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12479","reported_seq":12607,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.960738+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:24.960738+0000","last_peered":"2024-09-11T05:07:24.960738+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:24.960738+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"28'2400","ondisk_log_start":"28'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2525504,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12476,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12312","reported_seq":12429,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.722952+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:29.722952+0000","last_peered":"2024-09-11T05:07:29.722952+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:29.722952+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1766560,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12310,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12331","reported_seq":12452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.928925+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:24.928925+0000","last_peered":"2024-09-11T05:07:24.928925+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:24.928925+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2633456,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12100","reported_seq":16706,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:26.390230+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:26.390230+0000","last_peered":"2024-09-11T05:07:26.390230+0000","last_clean":"2024-09-11T05:07:26.390230+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:26.390230+0000","last_undegraded":"2024-09-11T05:07:26.390230+0000","last_fullsized":"2024-09-11T05:07:26.390230+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2326528,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12420","reported_seq":17202,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:26.258792+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:26.258792+0000","last_peered":"2024-09-11T05:07:26.258792+0000","last_clean":"2024-09-11T05:07:26.258792+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:26.258792+0000","last_undegraded":"2024-09-11T05:07:26.258792+0000","last_fullsized":"2024-09-11T05:07:26.258792+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2473984,"num_objects":302,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12268","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:24.888092+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:24.888092+0000","last_peered":"2024-09-11T05:07:24.888092+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:24.888092+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2651296,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12266,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12183","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.605442+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:29.605442+0000","last_peered":"2024-09-11T05:07:29.605442+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:29.605442+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1848480,"num_objects":226,"num_object_clones":0,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12181,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2077}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12488","reported_seq":12612,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.805286+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:26.805286+0000","last_peered":"2024-09-11T05:07:26.805286+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:26.805286+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2364576,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12486,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":36960152,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169533,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":72634368,"data_stored":72634368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160789,"ondisk_log_size":160789,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723340,"kb_used_data":19708,"kb_used_omap":4,"kb_used_meta":703611,"kb_avail":93648500,"statfs":{"total":96636764160,"available":95896064000,"internally_reserved":0,"allocated":20180992,"data_stored":19982483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":720498384},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701664,"kb_used_data":16592,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93670176,"statfs":{"total":96636764160,"available":95918260224,"internally_reserved":0,"allocated":16990208,"data_stored":16791699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587556,"kb_used_data":18644,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93784284,"statfs":{"total":96636764160,"available":96035106816,"internally_reserved":0,"allocated":19091456,"data_stored":18892947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582545106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699884,"kb_used_data":18524,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93671956,"statfs":{"total":96636764160,"available":95920082944,"internally_reserved":0,"allocated":18968576,"data_stored":18770067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,12,3,2,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18235392,"data_stored":18235392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18690048,"data_stored":18690048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15945728,"data_stored":15945728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19763200,"data_stored":19763200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:31.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:31.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12208", 'reported_seq': 12307, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:24.987180+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:24.987180+0000', 'last_peered': '2024-09-11T05:07:24.987180+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:24.987180+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2506752, 'num_objects': 306, 'num_object_clones': 0, 'num_object_copies': 1224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12208, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:31.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:31.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12465", 'reported_seq': 12585, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:26.946844+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:26.946844+0000', 'last_peered': '2024-09-11T05:07:26.946844+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:26.946844+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2233504, 'num_objects': 273, 'num_object_clones': 0, 'num_object_copies': 1092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12463, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:31.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:31.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11927", 'reported_seq': 12036, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:29.768273+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:29.768273+0000', 'last_peered': '2024-09-11T05:07:29.768273+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:29.768273+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1907304, 'num_objects': 234, 'num_object_clones': 0, 'num_object_copies': 936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 234, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11926, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:31.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12132", 'reported_seq': 13204, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:27.045649+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:27.045649+0000', 'last_peered': '2024-09-11T05:07:27.045649+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:27.045649+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2323616, 'num_objects': 284, 'num_object_clones': 0, 'num_object_copies': 1136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1784, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12130, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 498, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1500}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12239", 'reported_seq': 12367, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:24.867700+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:24.867700+0000', 'last_peered': '2024-09-11T05:07:24.867700+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:24.867700+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2389152, 'num_objects': 292, 'num_object_clones': 0, 'num_object_copies': 1168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12237, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1929}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12338", 'reported_seq': 12448, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:24.888535+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:24.888535+0000', 'last_peered': '2024-09-11T05:07:24.888535+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:24.888535+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2473984, 'num_objects': 302, 'num_object_clones': 0, 'num_object_copies': 1208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12338, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1992}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12463", 'reported_seq': 12562, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:26.875593+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:26.875593+0000', 'last_peered': '2024-09-11T05:07:26.875593+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:26.875593+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2184352, 'num_objects': 267, 'num_object_clones': 0, 'num_object_copies': 1068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12461, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12479", 'reported_seq': 12607, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:24.960738+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:24.960738+0000', 'last_peered': '2024-09-11T05:07:24.960738+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:24.960738+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "28'2400", 'ondisk_log_start': "28'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2525504, 'num_objects': 309, 'num_object_clones': 0, 'num_object_copies': 1236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12476, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:31.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12312", 'reported_seq': 12429, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:29.722952+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:29.722952+0000', 'last_peered': '2024-09-11T05:07:29.722952+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:29.722952+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1766560, 'num_objects': 216, 'num_object_clones': 0, 'num_object_copies': 864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12310, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12331", 'reported_seq': 12452, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:24.928925+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:24.928925+0000', 'last_peered': '2024-09-11T05:07:24.928925+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:24.928925+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2633456, 'num_objects': 322, 'num_object_clones': 0, 'num_object_copies': 1288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12328, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1976}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12268", 'reported_seq': 12380, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:24.888092+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:24.888092+0000', 'last_peered': '2024-09-11T05:07:24.888092+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:24.888092+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2651296, 'num_objects': 324, 'num_object_clones': 0, 'num_object_copies': 1296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12266, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12183", 'reported_seq': 12307, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:29.605442+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:29.605442+0000', 'last_peered': '2024-09-11T05:07:29.605442+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:29.605442+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1848480, 'num_objects': 226, 'num_object_clones': 0, 'num_object_copies': 904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12181, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2077}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:31.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12488", 'reported_seq': 12612, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:26.805286+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:26.805286+0000', 'last_peered': '2024-09-11T05:07:26.805286+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:26.805286+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2364576, 'num_objects': 289, 'num_object_clones': 0, 'num_object_copies': 1156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12486, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1999}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:31.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:32.545 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:32.545 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29212/15868 objects degraded (184.094%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":779,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":3967,"data_bytes":32454552,"bytes_used":2778775552,"bytes_avail":383768281088,"bytes_total":386547056640,"degraded_objects":29212,"degraded_total":15868,"degraded_ratio":1.8409377363246788,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":179},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:32.568 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:32.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:33.214 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:33.219 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:33.247 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-11T05:07:31.924697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":32454552,"num_objects":3967,"num_object_clones":0,"num_object_copies":15868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170083,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2713648,"kb_used_data":66352,"kb_used_omap":14,"kb_used_meta":2647217,"kb_avail":374773712,"statfs":{"total":386547056640,"available":383768281088,"internally_reserved":0,"allocated":67944448,"data_stored":67150412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2710750774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,8,16,11,12,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17661952,"num_objects":-2156,"num_object_clones":0,"num_object_copies":-8624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007575"},"pg_stats":[{"pgid":"1.f","version":"37'12275","reported_seq":12374,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:30.026393+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:30.026393+0000","last_peered":"2024-09-11T05:07:30.026393+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:30.026393+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1957888,"num_objects":239,"num_object_clones":0,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12275,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12465","reported_seq":12585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.946844+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:26.946844+0000","last_peered":"2024-09-11T05:07:26.946844+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:26.946844+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2233504,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12463,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11927","reported_seq":12036,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.768273+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:29.768273+0000","last_peered":"2024-09-11T05:07:29.768273+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:29.768273+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1907304,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11926,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12132","reported_seq":13204,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:27.045649+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:27.045649+0000","last_peered":"2024-09-11T05:07:27.045649+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:27.045649+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2323616,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12130,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12296","reported_seq":12424,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.795152+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:29.795152+0000","last_peered":"2024-09-11T05:07:29.795152+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:29.795152+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1922208,"num_objects":235,"num_object_clones":0,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12294,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12405","reported_seq":12516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.917698+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:29.917698+0000","last_peered":"2024-09-11T05:07:29.917698+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:29.917698+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1925120,"num_objects":235,"num_object_clones":0,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12463","reported_seq":12562,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.875593+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:26.875593+0000","last_peered":"2024-09-11T05:07:26.875593+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:26.875593+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2184352,"num_objects":267,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2040}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12592","reported_seq":17455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.268028+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:31.268028+0000","last_peered":"2024-09-11T05:07:31.268028+0000","last_clean":"2024-09-11T05:07:31.268028+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:31.268028+0000","last_undegraded":"2024-09-11T05:07:31.268028+0000","last_fullsized":"2024-09-11T05:07:31.268028+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1892352,"num_objects":231,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12541","reported_seq":12669,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.768179+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:29.768179+0000","last_peered":"2024-09-11T05:07:29.768179+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:29.768179+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2017600,"num_objects":247,"num_object_clones":0,"num_object_copies":988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12312","reported_seq":12429,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.722952+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:29.722952+0000","last_peered":"2024-09-11T05:07:29.722952+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:29.722952+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1766560,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12310,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12391","reported_seq":12511,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:30.000511+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:30.000511+0000","last_peered":"2024-09-11T05:07:30.000511+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:30.000511+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2141936,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12388,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12167","reported_seq":16773,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.498793+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:31.498793+0000","last_peered":"2024-09-11T05:07:31.498793+0000","last_clean":"2024-09-11T05:07:31.498793+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:31.498793+0000","last_undegraded":"2024-09-11T05:07:31.498793+0000","last_fullsized":"2024-09-11T05:07:31.498793+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1777664,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12480","reported_seq":17262,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.387841+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:31.387841+0000","last_peered":"2024-09-11T05:07:31.387841+0000","last_clean":"2024-09-11T05:07:31.387841+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:31.387841+0000","last_undegraded":"2024-09-11T05:07:31.387841+0000","last_fullsized":"2024-09-11T05:07:31.387841+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1982464,"num_objects":242,"num_object_clones":0,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":627,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12322","reported_seq":12434,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.879943+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:29.879943+0000","last_peered":"2024-09-11T05:07:29.879943+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:29.879943+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2208928,"num_objects":270,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12320,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1994}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12183","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:29.605442+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:29.605442+0000","last_peered":"2024-09-11T05:07:29.605442+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:29.605442+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1848480,"num_objects":226,"num_object_clones":0,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12181,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2077}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12488","reported_seq":12612,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:26.805286+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:26.805286+0000","last_peered":"2024-09-11T05:07:26.805286+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:26.805286+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2364576,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12486,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":32454552,"num_objects":3967,"num_object_clones":0,"num_object_copies":15868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170083,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64634880,"data_stored":64634880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724300,"kb_used_data":16508,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93647540,"statfs":{"total":96636764160,"available":95895080960,"internally_reserved":0,"allocated":16904192,"data_stored":16705683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701664,"kb_used_data":16592,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93670176,"statfs":{"total":96636764160,"available":95918260224,"internally_reserved":0,"allocated":16990208,"data_stored":16791699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587800,"kb_used_data":14728,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93784040,"statfs":{"total":96636764160,"available":96034856960,"internally_reserved":0,"allocated":15081472,"data_stored":14882963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586804946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,6,6,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699884,"kb_used_data":18524,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93671956,"statfs":{"total":96636764160,"available":95920082944,"internally_reserved":0,"allocated":18968576,"data_stored":18770067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,12,3,2,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18235392,"data_stored":18235392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14667776,"data_stored":14667776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15945728,"data_stored":15945728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15785984,"data_stored":15785984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:36.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:36.916 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:36.917 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:36.948 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-11T05:07:35.925708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25999256,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170871,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161027,"ondisk_log_size":161027,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2733464,"kb_used_data":54228,"kb_used_omap":14,"kb_used_meta":2679153,"kb_avail":374753896,"statfs":{"total":386547056640,"available":383747989504,"internally_reserved":0,"allocated":55529472,"data_stored":54735436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743453238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,15,7,9,17,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20324352,"num_objects":-2481,"num_object_clones":0,"num_object_copies":-9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2481,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003808"},"pg_stats":[{"pgid":"1.f","version":"37'12336","reported_seq":12436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.156679+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:35.156679+0000","last_peered":"2024-09-11T05:07:35.156679+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:35.156679+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1458176,"num_objects":178,"num_object_clones":0,"num_object_copies":712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12528","reported_seq":12646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:32.186769+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:32.186769+0000","last_peered":"2024-09-11T05:07:32.186769+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:32.186769+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1717408,"num_objects":210,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12526,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11986","reported_seq":12096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.818365+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:34.818365+0000","last_peered":"2024-09-11T05:07:34.818365+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:34.818365+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1423976,"num_objects":175,"num_object_clones":0,"num_object_copies":700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11985,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12191","reported_seq":13356,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:32.169139+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:32.169139+0000","last_peered":"2024-09-11T05:07:32.169139+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:32.169139+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1840288,"num_objects":225,"num_object_clones":0,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12189,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12350","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.056437+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:35.056437+0000","last_peered":"2024-09-11T05:07:35.056437+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:35.056437+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1479840,"num_objects":181,"num_object_clones":0,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12348,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2040}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12475","reported_seq":12585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.107371+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:35.107371+0000","last_peered":"2024-09-11T05:07:35.107371+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:35.107371+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1351680,"num_objects":165,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12475,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12520","reported_seq":12619,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:32.072033+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:32.072033+0000","last_peered":"2024-09-11T05:07:32.072033+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:32.072033+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1717408,"num_objects":210,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12592","reported_seq":17455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.268028+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:31.268028+0000","last_peered":"2024-09-11T05:07:31.268028+0000","last_clean":"2024-09-11T05:07:31.268028+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:31.268028+0000","last_undegraded":"2024-09-11T05:07:31.268028+0000","last_fullsized":"2024-09-11T05:07:31.268028+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1892352,"num_objects":231,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12589","reported_seq":12716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.735358+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:34.735358+0000","last_peered":"2024-09-11T05:07:34.735358+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:34.735358+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1624384,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12375","reported_seq":12493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.884131+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:34.884131+0000","last_peered":"2024-09-11T05:07:34.884131+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:34.884131+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12459","reported_seq":12581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.011451+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:35.011451+0000","last_peered":"2024-09-11T05:07:35.011451+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:35.011451+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1584880,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12167","reported_seq":16773,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.498793+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:31.498793+0000","last_peered":"2024-09-11T05:07:31.498793+0000","last_clean":"2024-09-11T05:07:31.498793+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:31.498793+0000","last_undegraded":"2024-09-11T05:07:31.498793+0000","last_fullsized":"2024-09-11T05:07:31.498793+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1777664,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12480","reported_seq":17262,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.387841+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:31.387841+0000","last_peered":"2024-09-11T05:07:31.387841+0000","last_clean":"2024-09-11T05:07:31.387841+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:31.387841+0000","last_undegraded":"2024-09-11T05:07:31.387841+0000","last_fullsized":"2024-09-11T05:07:31.387841+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1982464,"num_objects":242,"num_object_clones":0,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":627,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12386","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.025044+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:35.025044+0000","last_peered":"2024-09-11T05:07:35.025044+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:35.025044+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1684640,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12384,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2058}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12239","reported_seq":12363,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.592379+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:34.592379+0000","last_peered":"2024-09-11T05:07:34.592379+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:34.592379+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1389728,"num_objects":170,"num_object_clones":0,"num_object_copies":680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2133}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12554","reported_seq":12679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:31.903022+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:31.903022+0000","last_peered":"2024-09-11T05:07:31.903022+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:31.903022+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1823904,"num_objects":223,"num_object_clones":0,"num_object_copies":892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12552,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2065}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":25999256,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170871,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":52727808,"data_stored":52727808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161027,"ondisk_log_size":161027,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725412,"kb_used_data":12048,"kb_used_omap":4,"kb_used_meta":713339,"kb_avail":93646428,"statfs":{"total":96636764160,"available":95893942272,"internally_reserved":0,"allocated":12337152,"data_stored":12138643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":730459856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,1,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720112,"kb_used_data":12832,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93651728,"statfs":{"total":96636764160,"available":95899369472,"internally_reserved":0,"allocated":13139968,"data_stored":12941459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587800,"kb_used_data":14728,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93784040,"statfs":{"total":96636764160,"available":96034856960,"internally_reserved":0,"allocated":15081472,"data_stored":14882963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586804946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,6,6,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700140,"kb_used_data":14620,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93671700,"statfs":{"total":96636764160,"available":95919820800,"internally_reserved":0,"allocated":14970880,"data_stored":14772371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,0,0,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14172160,"data_stored":14172160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14667776,"data_stored":14667776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12050432,"data_stored":12050432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11837440,"data_stored":11837440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:36.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:37.651 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:37.652 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:37.672 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-11T05:07:35.925708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25999256,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170871,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161027,"ondisk_log_size":161027,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2733464,"kb_used_data":54228,"kb_used_omap":14,"kb_used_meta":2679153,"kb_avail":374753896,"statfs":{"total":386547056640,"available":383747989504,"internally_reserved":0,"allocated":55529472,"data_stored":54735436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2743453238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,15,7,9,17,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20324352,"num_objects":-2481,"num_object_clones":0,"num_object_copies":-9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2481,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003808"},"pg_stats":[{"pgid":"1.f","version":"37'12336","reported_seq":12436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.156679+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:35.156679+0000","last_peered":"2024-09-11T05:07:35.156679+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:35.156679+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1458176,"num_objects":178,"num_object_clones":0,"num_object_copies":712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12528","reported_seq":12646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:32.186769+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:32.186769+0000","last_peered":"2024-09-11T05:07:32.186769+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:32.186769+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1717408,"num_objects":210,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12526,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11986","reported_seq":12096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.818365+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:34.818365+0000","last_peered":"2024-09-11T05:07:34.818365+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:34.818365+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1423976,"num_objects":175,"num_object_clones":0,"num_object_copies":700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11985,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12191","reported_seq":13356,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:32.169139+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:32.169139+0000","last_peered":"2024-09-11T05:07:32.169139+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:32.169139+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1840288,"num_objects":225,"num_object_clones":0,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12189,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12350","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.056437+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:35.056437+0000","last_peered":"2024-09-11T05:07:35.056437+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:35.056437+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1479840,"num_objects":181,"num_object_clones":0,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12348,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2040}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12475","reported_seq":12585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.107371+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:35.107371+0000","last_peered":"2024-09-11T05:07:35.107371+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:35.107371+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1351680,"num_objects":165,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12475,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12520","reported_seq":12619,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:32.072033+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:32.072033+0000","last_peered":"2024-09-11T05:07:32.072033+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:32.072033+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1717408,"num_objects":210,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12592","reported_seq":17455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.268028+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:31.268028+0000","last_peered":"2024-09-11T05:07:31.268028+0000","last_clean":"2024-09-11T05:07:31.268028+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:31.268028+0000","last_undegraded":"2024-09-11T05:07:31.268028+0000","last_fullsized":"2024-09-11T05:07:31.268028+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1892352,"num_objects":231,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12589","reported_seq":12716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.735358+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:34.735358+0000","last_peered":"2024-09-11T05:07:34.735358+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:34.735358+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1624384,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12375","reported_seq":12493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.884131+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:34.884131+0000","last_peered":"2024-09-11T05:07:34.884131+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:34.884131+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12459","reported_seq":12581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.011451+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:35.011451+0000","last_peered":"2024-09-11T05:07:35.011451+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:35.011451+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1584880,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12167","reported_seq":16773,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.498793+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:31.498793+0000","last_peered":"2024-09-11T05:07:31.498793+0000","last_clean":"2024-09-11T05:07:31.498793+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:31.498793+0000","last_undegraded":"2024-09-11T05:07:31.498793+0000","last_fullsized":"2024-09-11T05:07:31.498793+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1777664,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12480","reported_seq":17262,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:31.387841+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:31.387841+0000","last_peered":"2024-09-11T05:07:31.387841+0000","last_clean":"2024-09-11T05:07:31.387841+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:31.387841+0000","last_undegraded":"2024-09-11T05:07:31.387841+0000","last_fullsized":"2024-09-11T05:07:31.387841+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1982464,"num_objects":242,"num_object_clones":0,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":627,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12386","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.025044+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:35.025044+0000","last_peered":"2024-09-11T05:07:35.025044+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:35.025044+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1684640,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12384,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2058}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12239","reported_seq":12363,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.592379+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:34.592379+0000","last_peered":"2024-09-11T05:07:34.592379+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:34.592379+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1389728,"num_objects":170,"num_object_clones":0,"num_object_copies":680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2133}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12554","reported_seq":12679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:31.903022+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:31.903022+0000","last_peered":"2024-09-11T05:07:31.903022+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:31.903022+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1823904,"num_objects":223,"num_object_clones":0,"num_object_copies":892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12552,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2065}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":25999256,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170871,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":52727808,"data_stored":52727808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161027,"ondisk_log_size":161027,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725412,"kb_used_data":12048,"kb_used_omap":4,"kb_used_meta":713339,"kb_avail":93646428,"statfs":{"total":96636764160,"available":95893942272,"internally_reserved":0,"allocated":12337152,"data_stored":12138643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":730459856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,1,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720112,"kb_used_data":12832,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93651728,"statfs":{"total":96636764160,"available":95899369472,"internally_reserved":0,"allocated":13139968,"data_stored":12941459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587800,"kb_used_data":14728,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93784040,"statfs":{"total":96636764160,"available":96034856960,"internally_reserved":0,"allocated":15081472,"data_stored":14882963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586804946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,6,6,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700140,"kb_used_data":14620,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93671700,"statfs":{"total":96636764160,"available":95919820800,"internally_reserved":0,"allocated":14970880,"data_stored":14772371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,0,0,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14172160,"data_stored":14172160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14667776,"data_stored":14667776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12050432,"data_stored":12050432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11837440,"data_stored":11837440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:37.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:37.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12336", 'reported_seq': 12436, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:35.156679+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:35.156679+0000', 'last_peered': '2024-09-11T05:07:35.156679+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:35.156679+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1458176, 'num_objects': 178, 'num_object_clones': 0, 'num_object_copies': 712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12336, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2113}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:37.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:37.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12528", 'reported_seq': 12646, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:32.186769+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:32.186769+0000', 'last_peered': '2024-09-11T05:07:32.186769+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:32.186769+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1717408, 'num_objects': 210, 'num_object_clones': 0, 'num_object_copies': 840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12526, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2048}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:37.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:37.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11986", 'reported_seq': 12096, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:34.818365+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:34.818365+0000', 'last_peered': '2024-09-11T05:07:34.818365+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:34.818365+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1423976, 'num_objects': 175, 'num_object_clones': 0, 'num_object_copies': 700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 175, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11985, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2036}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:37.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:37.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12191", 'reported_seq': 13356, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:32.169139+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:32.169139+0000', 'last_peered': '2024-09-11T05:07:32.169139+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:32.169139+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1840288, 'num_objects': 225, 'num_object_clones': 0, 'num_object_copies': 900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1737, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12189, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 545, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12350", 'reported_seq': 12477, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:35.056437+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:35.056437+0000', 'last_peered': '2024-09-11T05:07:35.056437+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:35.056437+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1479840, 'num_objects': 181, 'num_object_clones': 0, 'num_object_copies': 724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12348, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12475", 'reported_seq': 12585, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:35.107371+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:35.107371+0000', 'last_peered': '2024-09-11T05:07:35.107371+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:35.107371+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1351680, 'num_objects': 165, 'num_object_clones': 0, 'num_object_copies': 660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12475, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2129}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12520", 'reported_seq': 12619, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:32.072033+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:32.072033+0000', 'last_peered': '2024-09-11T05:07:32.072033+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:32.072033+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1717408, 'num_objects': 210, 'num_object_clones': 0, 'num_object_copies': 840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12518, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2097}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12589", 'reported_seq': 12716, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:34.735358+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:34.735358+0000', 'last_peered': '2024-09-11T05:07:34.735358+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:34.735358+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1624384, 'num_objects': 199, 'num_object_clones': 0, 'num_object_copies': 796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12586, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2167}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:37.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12375", 'reported_seq': 12493, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:34.884131+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:34.884131+0000', 'last_peered': '2024-09-11T05:07:34.884131+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:34.884131+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1250464, 'num_objects': 153, 'num_object_clones': 0, 'num_object_copies': 612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12373, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2153}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:37.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:37.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12459", 'reported_seq': 12581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:35.011451+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:35.011451+0000', 'last_peered': '2024-09-11T05:07:35.011451+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:35.011451+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1584880, 'num_objects': 194, 'num_object_clones': 0, 'num_object_copies': 776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12456, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:37.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:37.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12386", 'reported_seq': 12498, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:35.025044+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:35.025044+0000', 'last_peered': '2024-09-11T05:07:35.025044+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:35.025044+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1684640, 'num_objects': 206, 'num_object_clones': 0, 'num_object_copies': 824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12384, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:37.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:37.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12239", 'reported_seq': 12363, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:34.592379+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:34.592379+0000', 'last_peered': '2024-09-11T05:07:34.592379+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:34.592379+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1389728, 'num_objects': 170, 'num_object_clones': 0, 'num_object_copies': 680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12237, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2133}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:37.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:37.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12554", 'reported_seq': 12679, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:31.903022+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:31.903022+0000', 'last_peered': '2024-09-11T05:07:31.903022+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:31.903022+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1823904, 'num_objects': 223, 'num_object_clones': 0, 'num_object_copies': 892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12552, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2065}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:37.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:38.488 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:38.488 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29114/11008 objects degraded (264.480%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":785,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2752,"data_bytes":22501272,"bytes_used":2818207744,"bytes_avail":383728848896,"bytes_total":386547056640,"degraded_objects":29114,"degraded_total":11008,"degraded_ratio":2.6448037790697674,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":197},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:38.513 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:38.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:39.155 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:39.187 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-11T05:07:37.926209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22501272,"num_objects":2752,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171298,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160954,"ondisk_log_size":160954,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2752156,"kb_used_data":46544,"kb_used_omap":14,"kb_used_meta":2705521,"kb_avail":374735204,"statfs":{"total":386547056640,"available":383728848896,"internally_reserved":0,"allocated":47661056,"data_stored":46867020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2770454070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,18,8,9,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19464192,"num_objects":-2376,"num_object_clones":0,"num_object_copies":-9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003354"},"pg_stats":[{"pgid":"1.f","version":"37'12336","reported_seq":12436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.156679+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:35.156679+0000","last_peered":"2024-09-11T05:07:35.156679+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:35.156679+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1458176,"num_objects":178,"num_object_clones":0,"num_object_copies":712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12585","reported_seq":12705,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.245878+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:37.245878+0000","last_peered":"2024-09-11T05:07:37.245878+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:37.245878+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12583,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11986","reported_seq":12096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.818365+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:34.818365+0000","last_peered":"2024-09-11T05:07:34.818365+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:34.818365+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1423976,"num_objects":175,"num_object_clones":0,"num_object_copies":700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11985,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12251","reported_seq":13519,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.276300+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:37.276300+0000","last_peered":"2024-09-11T05:07:37.276300+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:37.276300+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1348768,"num_objects":165,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12350","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.056437+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:35.056437+0000","last_peered":"2024-09-11T05:07:35.056437+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:35.056437+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1479840,"num_objects":181,"num_object_clones":0,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12348,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2040}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12475","reported_seq":12585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.107371+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:35.107371+0000","last_peered":"2024-09-11T05:07:35.107371+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:35.107371+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1351680,"num_objects":165,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12475,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12577","reported_seq":12676,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.114937+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:37.114937+0000","last_peered":"2024-09-11T05:07:37.114937+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:37.114937+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12575,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2154}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12652","reported_seq":17515,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:36.552762+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:36.552762+0000","last_peered":"2024-09-11T05:07:36.552762+0000","last_clean":"2024-09-11T05:07:36.552762+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:36.552762+0000","last_undegraded":"2024-09-11T05:07:36.552762+0000","last_fullsized":"2024-09-11T05:07:36.552762+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1400832,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12589","reported_seq":12716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.735358+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:34.735358+0000","last_peered":"2024-09-11T05:07:34.735358+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:34.735358+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1624384,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12375","reported_seq":12493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.884131+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:34.884131+0000","last_peered":"2024-09-11T05:07:34.884131+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:34.884131+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12459","reported_seq":12581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.011451+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:35.011451+0000","last_peered":"2024-09-11T05:07:35.011451+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:35.011451+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1584880,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12456,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12237","reported_seq":16842,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:36.711443+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:36.711443+0000","last_peered":"2024-09-11T05:07:36.711443+0000","last_clean":"2024-09-11T05:07:36.711443+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:36.711443+0000","last_undegraded":"2024-09-11T05:07:36.711443+0000","last_fullsized":"2024-09-11T05:07:36.711443+0000","mapping_epoch":32,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1204224,"num_objects":147,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6047,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12549","reported_seq":17331,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:36.432396+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:36.432396+0000","last_peered":"2024-09-11T05:07:36.432396+0000","last_clean":"2024-09-11T05:07:36.432396+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:36.432396+0000","last_undegraded":"2024-09-11T05:07:36.432396+0000","last_fullsized":"2024-09-11T05:07:36.432396+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1417216,"num_objects":173,"num_object_clones":0,"num_object_copies":692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":696,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12386","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:35.025044+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:35.025044+0000","last_peered":"2024-09-11T05:07:35.025044+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:35.025044+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1684640,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12384,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2058}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12239","reported_seq":12363,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:34.592379+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:34.592379+0000","last_peered":"2024-09-11T05:07:34.592379+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:34.592379+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1389728,"num_objects":170,"num_object_clones":0,"num_object_copies":680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2133}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12608","reported_seq":12733,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.128868+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:37.128868+0000","last_peered":"2024-09-11T05:07:37.128868+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:37.128868+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1381536,"num_objects":169,"num_object_clones":0,"num_object_copies":676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2119}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":22501272,"num_objects":2752,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171298,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44789760,"data_stored":44789760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160954,"ondisk_log_size":160954,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725412,"kb_used_data":12048,"kb_used_omap":4,"kb_used_meta":713339,"kb_avail":93646428,"statfs":{"total":96636764160,"available":95893942272,"internally_reserved":0,"allocated":12337152,"data_stored":12138643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":730459856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,1,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720112,"kb_used_data":12832,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93651728,"statfs":{"total":96636764160,"available":95899369472,"internally_reserved":0,"allocated":13139968,"data_stored":12941459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588032,"kb_used_data":10792,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93783808,"statfs":{"total":96636764160,"available":96034619392,"internally_reserved":0,"allocated":11051008,"data_stored":10852499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591064786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10,0,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718600,"kb_used_data":10872,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93653240,"statfs":{"total":96636764160,"available":95900917760,"internally_reserved":0,"allocated":11132928,"data_stored":10934419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":724692687},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10211328,"data_stored":10211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10690560,"data_stored":10690560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12050432,"data_stored":12050432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11837440,"data_stored":11837440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:42.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:42.795 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:42.796 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:42.827 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-11T05:07:41.927157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16766872,"num_objects":2052,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171998,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735804,"kb_used_data":34336,"kb_used_omap":14,"kb_used_meta":2701361,"kb_avail":374751556,"statfs":{"total":386547056640,"available":383745593344,"internally_reserved":0,"allocated":35160064,"data_stored":34366028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766194230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,5,15,13,11,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20193280,"num_objects":-2465,"num_object_clones":0,"num_object_copies":-9860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2465,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002939"},"pg_stats":[{"pgid":"1.f","version":"37'12390","reported_seq":12490,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.205455+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:40.205455+0000","last_peered":"2024-09-11T05:07:40.205455+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:40.205455+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015808,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12390,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12585","reported_seq":12705,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.245878+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:37.245878+0000","last_peered":"2024-09-11T05:07:37.245878+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:37.245878+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12583,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12039","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.861050+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:39.861050+0000","last_peered":"2024-09-11T05:07:39.861050+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:39.861050+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":989800,"num_objects":122,"num_object_clones":0,"num_object_copies":488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12038,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12251","reported_seq":13519,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.276300+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:37.276300+0000","last_peered":"2024-09-11T05:07:37.276300+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:37.276300+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1348768,"num_objects":165,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12404","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.116902+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:40.116902+0000","last_peered":"2024-09-11T05:07:40.116902+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:40.116902+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1037472,"num_objects":127,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12402,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12526","reported_seq":12637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.734152+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:39.734152+0000","last_peered":"2024-09-11T05:07:39.734152+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:39.734152+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933888,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12526,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12577","reported_seq":12676,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.114937+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:37.114937+0000","last_peered":"2024-09-11T05:07:37.114937+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:37.114937+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12575,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2154}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12718","reported_seq":17581,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.813041+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:41.813041+0000","last_peered":"2024-09-11T05:07:41.813041+0000","last_clean":"2024-09-11T05:07:41.813041+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:41.813041+0000","last_undegraded":"2024-09-11T05:07:41.813041+0000","last_fullsized":"2024-09-11T05:07:41.813041+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":860160,"num_objects":105,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12649","reported_seq":12776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.169959+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:40.169959+0000","last_peered":"2024-09-11T05:07:40.169959+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:40.169959+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1132864,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12430","reported_seq":12548,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.915629+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:39.915629+0000","last_peered":"2024-09-11T05:07:39.915629+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:39.915629+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":799904,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2208}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12532","reported_seq":12652,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.113736+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:40.113736+0000","last_peered":"2024-09-11T05:07:40.113736+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:40.113736+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":986864,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12529,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12301","reported_seq":16907,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.748206+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:41.748206+0000","last_peered":"2024-09-11T05:07:41.748206+0000","last_clean":"2024-09-11T05:07:41.748206+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:41.748206+0000","last_undegraded":"2024-09-11T05:07:41.748206+0000","last_fullsized":"2024-09-11T05:07:41.748206+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":679936,"num_objects":83,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12610","reported_seq":17392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.742096+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:41.742096+0000","last_peered":"2024-09-11T05:07:41.742096+0000","last_clean":"2024-09-11T05:07:41.742096+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:41.742096+0000","last_undegraded":"2024-09-11T05:07:41.742096+0000","last_fullsized":"2024-09-11T05:07:41.742096+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":112,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12446","reported_seq":12558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.081883+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:40.081883+0000","last_peered":"2024-09-11T05:07:40.081883+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:40.081883+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1193120,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12444,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12288","reported_seq":12412,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.688035+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:39.688035+0000","last_peered":"2024-09-11T05:07:39.688035+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:39.688035+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988320,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2182}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12608","reported_seq":12733,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.128868+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:37.128868+0000","last_peered":"2024-09-11T05:07:37.128868+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:37.128868+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1381536,"num_objects":169,"num_object_clones":0,"num_object_copies":676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2119}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":16766872,"num_objects":2052,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171998,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32956416,"data_stored":32956416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725732,"kb_used_data":8212,"kb_used_omap":4,"kb_used_meta":717499,"kb_avail":93646108,"statfs":{"total":96636764160,"available":95893614592,"internally_reserved":0,"allocated":8409088,"data_stored":8210579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":734719696},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,3,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703080,"kb_used_data":8260,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93668760,"statfs":{"total":96636764160,"available":95916810240,"internally_reserved":0,"allocated":8458240,"data_stored":8259731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,7,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588392,"kb_used_data":6992,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93783448,"statfs":{"total":96636764160,"available":96034250752,"internally_reserved":0,"allocated":7159808,"data_stored":6961299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,5,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718600,"kb_used_data":10872,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93653240,"statfs":{"total":96636764160,"available":95900917760,"internally_reserved":0,"allocated":11132928,"data_stored":10934419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":724692687},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10211328,"data_stored":10211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6676480,"data_stored":6676480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8114176,"data_stored":8114176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7954432,"data_stored":7954432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:42.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:43.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:43.516 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:43.545 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-11T05:07:41.927157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16766872,"num_objects":2052,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171998,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735804,"kb_used_data":34336,"kb_used_omap":14,"kb_used_meta":2701361,"kb_avail":374751556,"statfs":{"total":386547056640,"available":383745593344,"internally_reserved":0,"allocated":35160064,"data_stored":34366028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2766194230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,5,15,13,11,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20193280,"num_objects":-2465,"num_object_clones":0,"num_object_copies":-9860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2465,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002939"},"pg_stats":[{"pgid":"1.f","version":"37'12390","reported_seq":12490,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.205455+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:40.205455+0000","last_peered":"2024-09-11T05:07:40.205455+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:40.205455+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015808,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12390,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12585","reported_seq":12705,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.245878+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:37.245878+0000","last_peered":"2024-09-11T05:07:37.245878+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:37.245878+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12583,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12039","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.861050+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:39.861050+0000","last_peered":"2024-09-11T05:07:39.861050+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:39.861050+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":989800,"num_objects":122,"num_object_clones":0,"num_object_copies":488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12038,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12251","reported_seq":13519,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.276300+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:37.276300+0000","last_peered":"2024-09-11T05:07:37.276300+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:37.276300+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1348768,"num_objects":165,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12404","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.116902+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:40.116902+0000","last_peered":"2024-09-11T05:07:40.116902+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:40.116902+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1037472,"num_objects":127,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12402,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12526","reported_seq":12637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.734152+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:39.734152+0000","last_peered":"2024-09-11T05:07:39.734152+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:39.734152+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933888,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12526,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12577","reported_seq":12676,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.114937+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:37.114937+0000","last_peered":"2024-09-11T05:07:37.114937+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:37.114937+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250464,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12575,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2154}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12718","reported_seq":17581,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.813041+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:41.813041+0000","last_peered":"2024-09-11T05:07:41.813041+0000","last_clean":"2024-09-11T05:07:41.813041+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:41.813041+0000","last_undegraded":"2024-09-11T05:07:41.813041+0000","last_fullsized":"2024-09-11T05:07:41.813041+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":860160,"num_objects":105,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12649","reported_seq":12776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.169959+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:40.169959+0000","last_peered":"2024-09-11T05:07:40.169959+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:40.169959+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1132864,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12430","reported_seq":12548,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.915629+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:39.915629+0000","last_peered":"2024-09-11T05:07:39.915629+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:39.915629+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":799904,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2208}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12532","reported_seq":12652,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.113736+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:40.113736+0000","last_peered":"2024-09-11T05:07:40.113736+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:40.113736+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":986864,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12529,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12301","reported_seq":16907,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.748206+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:41.748206+0000","last_peered":"2024-09-11T05:07:41.748206+0000","last_clean":"2024-09-11T05:07:41.748206+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:41.748206+0000","last_undegraded":"2024-09-11T05:07:41.748206+0000","last_fullsized":"2024-09-11T05:07:41.748206+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":679936,"num_objects":83,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12610","reported_seq":17392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.742096+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:41.742096+0000","last_peered":"2024-09-11T05:07:41.742096+0000","last_clean":"2024-09-11T05:07:41.742096+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:41.742096+0000","last_undegraded":"2024-09-11T05:07:41.742096+0000","last_fullsized":"2024-09-11T05:07:41.742096+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":112,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12446","reported_seq":12558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.081883+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:40.081883+0000","last_peered":"2024-09-11T05:07:40.081883+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:40.081883+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1193120,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12444,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12288","reported_seq":12412,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.688035+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:39.688035+0000","last_peered":"2024-09-11T05:07:39.688035+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:39.688035+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988320,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2182}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12608","reported_seq":12733,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:37.128868+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:37.128868+0000","last_peered":"2024-09-11T05:07:37.128868+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:37.128868+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1381536,"num_objects":169,"num_object_clones":0,"num_object_copies":676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2119}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":16766872,"num_objects":2052,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171998,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32956416,"data_stored":32956416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725732,"kb_used_data":8212,"kb_used_omap":4,"kb_used_meta":717499,"kb_avail":93646108,"statfs":{"total":96636764160,"available":95893614592,"internally_reserved":0,"allocated":8409088,"data_stored":8210579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":734719696},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,3,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703080,"kb_used_data":8260,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93668760,"statfs":{"total":96636764160,"available":95916810240,"internally_reserved":0,"allocated":8458240,"data_stored":8259731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,7,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588392,"kb_used_data":6992,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93783448,"statfs":{"total":96636764160,"available":96034250752,"internally_reserved":0,"allocated":7159808,"data_stored":6961299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,5,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718600,"kb_used_data":10872,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93653240,"statfs":{"total":96636764160,"available":95900917760,"internally_reserved":0,"allocated":11132928,"data_stored":10934419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":724692687},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10211328,"data_stored":10211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6676480,"data_stored":6676480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8114176,"data_stored":8114176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7954432,"data_stored":7954432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:43.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:43.546 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12390", 'reported_seq': 12490, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:40.205455+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:40.205455+0000', 'last_peered': '2024-09-11T05:07:40.205455+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:40.205455+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1015808, 'num_objects': 124, 'num_object_clones': 0, 'num_object_copies': 496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12390, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2167}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:43.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:43.546 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12585", 'reported_seq': 12705, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:37.245878+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:37.245878+0000', 'last_peered': '2024-09-11T05:07:37.245878+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:37.245878+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1250464, 'num_objects': 153, 'num_object_clones': 0, 'num_object_copies': 612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12583, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12039", 'reported_seq': 12149, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:39.861050+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:39.861050+0000', 'last_peered': '2024-09-11T05:07:39.861050+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:39.861050+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 989800, 'num_objects': 122, 'num_object_clones': 0, 'num_object_copies': 488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 122, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12038, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2089}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12251", 'reported_seq': 13519, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:37.276300+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:37.276300+0000', 'last_peered': '2024-09-11T05:07:37.276300+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:37.276300+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1348768, 'num_objects': 165, 'num_object_clones': 0, 'num_object_copies': 660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1686, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12249, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 596, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12404", 'reported_seq': 12532, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:40.116902+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:40.116902+0000', 'last_peered': '2024-09-11T05:07:40.116902+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:40.116902+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1037472, 'num_objects': 127, 'num_object_clones': 0, 'num_object_copies': 508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12402, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2094}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12526", 'reported_seq': 12637, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:39.734152+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:39.734152+0000', 'last_peered': '2024-09-11T05:07:39.734152+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:39.734152+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 933888, 'num_objects': 114, 'num_object_clones': 0, 'num_object_copies': 456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12526, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2180}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:43.547 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12577", 'reported_seq': 12676, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:37.114937+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:37.114937+0000', 'last_peered': '2024-09-11T05:07:37.114937+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:37.114937+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1250464, 'num_objects': 153, 'num_object_clones': 0, 'num_object_copies': 612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12575, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2154}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:43.548 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:43.548 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12649", 'reported_seq': 12776, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:40.169959+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:40.169959+0000', 'last_peered': '2024-09-11T05:07:40.169959+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:40.169959+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1132864, 'num_objects': 139, 'num_object_clones': 0, 'num_object_copies': 556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12646, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2227}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:43.548 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:43.548 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12430", 'reported_seq': 12548, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:39.915629+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:39.915629+0000', 'last_peered': '2024-09-11T05:07:39.915629+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:39.915629+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 799904, 'num_objects': 98, 'num_object_clones': 0, 'num_object_copies': 392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 98, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2208}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:43.548 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:43.549 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12532", 'reported_seq': 12652, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:40.113736+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:40.113736+0000', 'last_peered': '2024-09-11T05:07:40.113736+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:40.113736+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 986864, 'num_objects': 121, 'num_object_clones': 0, 'num_object_copies': 484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12529, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:43.549 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:43.549 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12446", 'reported_seq': 12558, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:40.081883+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:40.081883+0000', 'last_peered': '2024-09-11T05:07:40.081883+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:40.081883+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1193120, 'num_objects': 146, 'num_object_clones': 0, 'num_object_copies': 584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12444, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:43.549 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:43.549 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12288", 'reported_seq': 12412, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:39.688035+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:39.688035+0000', 'last_peered': '2024-09-11T05:07:39.688035+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:39.688035+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 988320, 'num_objects': 121, 'num_object_clones': 0, 'num_object_copies': 484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12286, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2182}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:43.550 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:43.550 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12608", 'reported_seq': 12733, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:37.128868+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:37.128868+0000', 'last_peered': '2024-09-11T05:07:37.128868+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:37.128868+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1381536, 'num_objects': 169, 'num_object_clones': 0, 'num_object_copies': 676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12606, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2119}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:43.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:44.316 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29068/7216 objects degraded (402.827%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":791,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1804,"data_bytes":14735256,"bytes_used":2784768000,"bytes_avail":383762288640,"bytes_total":386547056640,"degraded_objects":29068,"degraded_total":7216,"degraded_ratio":4.0282705099778271,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:44.341 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:44.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:45.014 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:45.039 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-11T05:07:43.927724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":14735256,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172246,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160602,"ondisk_log_size":160602,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2719500,"kb_used_data":30512,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374767860,"statfs":{"total":386547056640,"available":383762288640,"internally_reserved":0,"allocated":31244288,"data_stored":30450252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2753414710},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,5,14,12,9,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17719296,"num_objects":-2163,"num_object_clones":0,"num_object_copies":-8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"1.f","version":"37'12390","reported_seq":12490,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.205455+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:40.205455+0000","last_peered":"2024-09-11T05:07:40.205455+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:40.205455+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015808,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12390,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12652","reported_seq":12772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:42.028480+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:42.028480+0000","last_peered":"2024-09-11T05:07:42.028480+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:42.028480+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":701600,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12039","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.861050+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:39.861050+0000","last_peered":"2024-09-11T05:07:39.861050+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:39.861050+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":989800,"num_objects":122,"num_object_clones":0,"num_object_copies":488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12038,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12316","reported_seq":13677,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:42.307136+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:42.307136+0000","last_peered":"2024-09-11T05:07:42.307136+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:42.307136+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":816288,"num_objects":100,"num_object_clones":0,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12314,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1540}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12404","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.116902+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:40.116902+0000","last_peered":"2024-09-11T05:07:40.116902+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:40.116902+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1037472,"num_objects":127,"num_object_clones":0,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12402,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12526","reported_seq":12637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.734152+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:39.734152+0000","last_peered":"2024-09-11T05:07:39.734152+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:39.734152+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933888,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12526,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12631","reported_seq":12730,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:42.283480+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:42.283480+0000","last_peered":"2024-09-11T05:07:42.283480+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:42.283480+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":808096,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12629,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2208}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12718","reported_seq":17581,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.813041+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:41.813041+0000","last_peered":"2024-09-11T05:07:41.813041+0000","last_clean":"2024-09-11T05:07:41.813041+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:41.813041+0000","last_undegraded":"2024-09-11T05:07:41.813041+0000","last_fullsized":"2024-09-11T05:07:41.813041+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":860160,"num_objects":105,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12649","reported_seq":12776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.169959+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:40.169959+0000","last_peered":"2024-09-11T05:07:40.169959+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:40.169959+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1132864,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12430","reported_seq":12548,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.915629+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:39.915629+0000","last_peered":"2024-09-11T05:07:39.915629+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:39.915629+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":799904,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2208}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12532","reported_seq":12652,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.113736+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:40.113736+0000","last_peered":"2024-09-11T05:07:40.113736+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:40.113736+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":986864,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12529,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12301","reported_seq":16907,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.748206+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:41.748206+0000","last_peered":"2024-09-11T05:07:41.748206+0000","last_clean":"2024-09-11T05:07:41.748206+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:41.748206+0000","last_undegraded":"2024-09-11T05:07:41.748206+0000","last_fullsized":"2024-09-11T05:07:41.748206+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":679936,"num_objects":83,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12610","reported_seq":17392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:41.742096+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:41.742096+0000","last_peered":"2024-09-11T05:07:41.742096+0000","last_clean":"2024-09-11T05:07:41.742096+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:41.742096+0000","last_undegraded":"2024-09-11T05:07:41.742096+0000","last_fullsized":"2024-09-11T05:07:41.742096+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":112,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":757,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12446","reported_seq":12558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:40.081883+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:40.081883+0000","last_peered":"2024-09-11T05:07:40.081883+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:40.081883+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1193120,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12444,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2118}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12288","reported_seq":12412,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:39.688035+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:39.688035+0000","last_peered":"2024-09-11T05:07:39.688035+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:39.688035+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988320,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2182}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12670","reported_seq":12795,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:42.239168+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:42.239168+0000","last_peered":"2024-09-11T05:07:42.239168+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:42.239168+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":873632,"num_objects":107,"num_object_clones":0,"num_object_copies":428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12668,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2181}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":14735256,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172246,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":29020160,"data_stored":29020160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160602,"ondisk_log_size":160602,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725732,"kb_used_data":8212,"kb_used_omap":4,"kb_used_meta":717499,"kb_avail":93646108,"statfs":{"total":96636764160,"available":95893614592,"internally_reserved":0,"allocated":8409088,"data_stored":8210579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":734719696},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,3,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703080,"kb_used_data":8260,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93668760,"statfs":{"total":96636764160,"available":95916810240,"internally_reserved":0,"allocated":8458240,"data_stored":8259731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,7,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588392,"kb_used_data":6992,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93783448,"statfs":{"total":96636764160,"available":96034250752,"internally_reserved":0,"allocated":7159808,"data_stored":6961299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,5,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702296,"kb_used_data":7048,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93669544,"statfs":{"total":96636764160,"available":95917613056,"internally_reserved":0,"allocated":7217152,"data_stored":7018643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":711913167},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6275072,"data_stored":6275072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6676480,"data_stored":6676480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8114176,"data_stored":8114176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7954432,"data_stored":7954432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:48.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:48.707 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:48.729 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-11T05:07:47.928774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6879128,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173205,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741140,"kb_used_data":16056,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374746220,"statfs":{"total":386547056640,"available":383740129280,"internally_reserved":0,"allocated":16441344,"data_stored":15647308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2790377014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,4,10,16,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19120128,"num_objects":-2334,"num_object_clones":0,"num_object_copies":-9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2334,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003065"},"pg_stats":[{"pgid":"1.f","version":"37'12446","reported_seq":12546,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.276977+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:45.276977+0000","last_peered":"2024-09-11T05:07:45.276977+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:45.276977+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":557056,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12446,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12706","reported_seq":12824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.275083+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:47.275083+0000","last_peered":"2024-09-11T05:07:47.275083+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:47.275083+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259232,"num_objects":32,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12704,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12104","reported_seq":12212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.036478+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:45.036478+0000","last_peered":"2024-09-11T05:07:45.036478+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:45.036478+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457320,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12103,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12365","reported_seq":13822,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.301570+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:47.301570+0000","last_peered":"2024-09-11T05:07:47.301570+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:47.301570+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414880,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12466","reported_seq":12594,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.257155+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:45.257155+0000","last_peered":"2024-09-11T05:07:45.257155+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:45.257155+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529568,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12464,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12580","reported_seq":12691,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.214878+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:45.214878+0000","last_peered":"2024-09-11T05:07:45.214878+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:45.214878+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12691","reported_seq":12790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.153966+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:47.153966+0000","last_peered":"2024-09-11T05:07:47.153966+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:47.153966+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316576,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12689,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12786","reported_seq":17649,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:46.883166+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:46.883166+0000","last_peered":"2024-09-11T05:07:46.883166+0000","last_clean":"2024-09-11T05:07:46.883166+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:46.883166+0000","last_undegraded":"2024-09-11T05:07:46.883166+0000","last_fullsized":"2024-09-11T05:07:46.883166+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303104,"num_objects":37,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12715","reported_seq":12843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.211719+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:45.211719+0000","last_peered":"2024-09-11T05:07:45.211719+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:45.211719+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":592192,"num_objects":73,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12712,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12483","reported_seq":12599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.058166+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:45.058166+0000","last_peered":"2024-09-11T05:07:45.058166+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:45.058166+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365728,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12591","reported_seq":12712,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.199773+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:45.199773+0000","last_peered":"2024-09-11T05:07:45.199773+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:45.199773+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503536,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12358","reported_seq":16964,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:47.026666+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:47.026666+0000","last_peered":"2024-09-11T05:07:47.026666+0000","last_clean":"2024-09-11T05:07:47.026666+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:47.026666+0000","last_undegraded":"2024-09-11T05:07:47.026666+0000","last_fullsized":"2024-09-11T05:07:47.026666+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212992,"num_objects":26,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6168,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12665","reported_seq":17446,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:47.003613+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:47.003613+0000","last_peered":"2024-09-11T05:07:47.003613+0000","last_clean":"2024-09-11T05:07:47.003613+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:47.003613+0000","last_undegraded":"2024-09-11T05:07:47.003613+0000","last_fullsized":"2024-09-11T05:07:47.003613+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":812,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12523","reported_seq":12635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.194694+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:45.194694+0000","last_peered":"2024-09-11T05:07:45.194694+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:45.194694+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562336,"num_objects":69,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2195}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12344","reported_seq":12468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:44.964392+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:44.964392+0000","last_peered":"2024-09-11T05:07:44.964392+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:44.964392+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529568,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12342,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12738","reported_seq":12863,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.331336+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:47.331336+0000","last_peered":"2024-09-11T05:07:47.331336+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:47.331336+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316576,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12736,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6879128,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173205,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13430784,"data_stored":13430784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726116,"kb_used_data":4412,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93645724,"statfs":{"total":96636764160,"available":95893221376,"internally_reserved":0,"allocated":4517888,"data_stored":4319379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":738979536},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722180,"kb_used_data":5172,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93649660,"statfs":{"total":96636764160,"available":95897251840,"internally_reserved":0,"allocated":5296128,"data_stored":5097619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,8,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572156,"kb_used_data":3240,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93799684,"statfs":{"total":96636764160,"available":96050876416,"internally_reserved":0,"allocated":3317760,"data_stored":3119251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582545106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720688,"kb_used_data":3232,"kb_used_omap":4,"kb_used_meta":717435,"kb_avail":93651152,"statfs":{"total":96636764160,"available":95898779648,"internally_reserved":0,"allocated":3309568,"data_stored":3111059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":734654159},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2342912,"data_stored":2342912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":2703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4300800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4083712,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:48.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:49.332 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:49.356 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-11T05:07:47.928774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6879128,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173205,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741140,"kb_used_data":16056,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374746220,"statfs":{"total":386547056640,"available":383740129280,"internally_reserved":0,"allocated":16441344,"data_stored":15647308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2790377014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,4,10,16,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19120128,"num_objects":-2334,"num_object_clones":0,"num_object_copies":-9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2334,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003065"},"pg_stats":[{"pgid":"1.f","version":"37'12446","reported_seq":12546,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.276977+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:45.276977+0000","last_peered":"2024-09-11T05:07:45.276977+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:45.276977+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":557056,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12446,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12706","reported_seq":12824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.275083+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:47.275083+0000","last_peered":"2024-09-11T05:07:47.275083+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:47.275083+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259232,"num_objects":32,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12704,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12104","reported_seq":12212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.036478+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:45.036478+0000","last_peered":"2024-09-11T05:07:45.036478+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:45.036478+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457320,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12103,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12365","reported_seq":13822,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.301570+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:47.301570+0000","last_peered":"2024-09-11T05:07:47.301570+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:47.301570+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414880,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12466","reported_seq":12594,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.257155+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:45.257155+0000","last_peered":"2024-09-11T05:07:45.257155+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:45.257155+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529568,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12464,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12580","reported_seq":12691,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.214878+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:45.214878+0000","last_peered":"2024-09-11T05:07:45.214878+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:45.214878+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12691","reported_seq":12790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.153966+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:47.153966+0000","last_peered":"2024-09-11T05:07:47.153966+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:47.153966+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316576,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12689,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12786","reported_seq":17649,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:46.883166+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:46.883166+0000","last_peered":"2024-09-11T05:07:46.883166+0000","last_clean":"2024-09-11T05:07:46.883166+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:46.883166+0000","last_undegraded":"2024-09-11T05:07:46.883166+0000","last_fullsized":"2024-09-11T05:07:46.883166+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303104,"num_objects":37,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12715","reported_seq":12843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.211719+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:45.211719+0000","last_peered":"2024-09-11T05:07:45.211719+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:45.211719+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":592192,"num_objects":73,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12712,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12483","reported_seq":12599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.058166+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:45.058166+0000","last_peered":"2024-09-11T05:07:45.058166+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:45.058166+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365728,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12591","reported_seq":12712,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.199773+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:45.199773+0000","last_peered":"2024-09-11T05:07:45.199773+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:45.199773+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503536,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12358","reported_seq":16964,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:47.026666+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:47.026666+0000","last_peered":"2024-09-11T05:07:47.026666+0000","last_clean":"2024-09-11T05:07:47.026666+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:47.026666+0000","last_undegraded":"2024-09-11T05:07:47.026666+0000","last_fullsized":"2024-09-11T05:07:47.026666+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212992,"num_objects":26,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6168,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12665","reported_seq":17446,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:47.003613+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:47.003613+0000","last_peered":"2024-09-11T05:07:47.003613+0000","last_clean":"2024-09-11T05:07:47.003613+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:47.003613+0000","last_undegraded":"2024-09-11T05:07:47.003613+0000","last_fullsized":"2024-09-11T05:07:47.003613+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":812,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12523","reported_seq":12635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.194694+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:45.194694+0000","last_peered":"2024-09-11T05:07:45.194694+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:45.194694+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562336,"num_objects":69,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2195}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12344","reported_seq":12468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:44.964392+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:44.964392+0000","last_peered":"2024-09-11T05:07:44.964392+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:44.964392+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529568,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12342,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12738","reported_seq":12863,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.331336+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:47.331336+0000","last_peered":"2024-09-11T05:07:47.331336+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:47.331336+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316576,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12736,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6879128,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173205,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13430784,"data_stored":13430784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726116,"kb_used_data":4412,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93645724,"statfs":{"total":96636764160,"available":95893221376,"internally_reserved":0,"allocated":4517888,"data_stored":4319379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":738979536},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722180,"kb_used_data":5172,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93649660,"statfs":{"total":96636764160,"available":95897251840,"internally_reserved":0,"allocated":5296128,"data_stored":5097619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,8,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572156,"kb_used_data":3240,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93799684,"statfs":{"total":96636764160,"available":96050876416,"internally_reserved":0,"allocated":3317760,"data_stored":3119251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582545106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720688,"kb_used_data":3232,"kb_used_omap":4,"kb_used_meta":717435,"kb_avail":93651152,"statfs":{"total":96636764160,"available":95898779648,"internally_reserved":0,"allocated":3309568,"data_stored":3111059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":734654159},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2342912,"data_stored":2342912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":2703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4300800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4083712,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:49.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:49.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12446", 'reported_seq': 12546, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:45.276977+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:45.276977+0000', 'last_peered': '2024-09-11T05:07:45.276977+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:45.276977+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 557056, 'num_objects': 68, 'num_object_clones': 0, 'num_object_copies': 272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 68, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12446, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2223}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12706", 'reported_seq': 12824, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:47.275083+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:47.275083+0000', 'last_peered': '2024-09-11T05:07:47.275083+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:47.275083+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 259232, 'num_objects': 32, 'num_object_clones': 0, 'num_object_copies': 128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 32, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12704, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12104", 'reported_seq': 12212, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:45.036478+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:45.036478+0000', 'last_peered': '2024-09-11T05:07:45.036478+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:45.036478+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 457320, 'num_objects': 57, 'num_object_clones': 0, 'num_object_copies': 228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 57, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12103, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2154}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12365", 'reported_seq': 13822, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:47.301570+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:47.301570+0000', 'last_peered': '2024-09-11T05:07:47.301570+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:47.301570+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 414880, 'num_objects': 51, 'num_object_clones': 0, 'num_object_copies': 204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1592, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 51, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12363, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 690, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1541}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12466", 'reported_seq': 12594, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:45.257155+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:45.257155+0000', 'last_peered': '2024-09-11T05:07:45.257155+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:45.257155+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 529568, 'num_objects': 65, 'num_object_clones': 0, 'num_object_copies': 260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 65, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12464, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2156}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:49.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12580", 'reported_seq': 12691, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:45.214878+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:45.214878+0000', 'last_peered': '2024-09-11T05:07:45.214878+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:45.214878+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 491520, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 60, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12580, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2234}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12691", 'reported_seq': 12790, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:47.153966+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:47.153966+0000', 'last_peered': '2024-09-11T05:07:47.153966+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:47.153966+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 316576, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12689, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2268}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12715", 'reported_seq': 12843, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:45.211719+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:45.211719+0000', 'last_peered': '2024-09-11T05:07:45.211719+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:45.211719+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 592192, 'num_objects': 73, 'num_object_clones': 0, 'num_object_copies': 292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 73, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12712, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2293}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12483", 'reported_seq': 12599, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:45.058166+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:45.058166+0000', 'last_peered': '2024-09-11T05:07:45.058166+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:45.058166+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 365728, 'num_objects': 45, 'num_object_clones': 0, 'num_object_copies': 180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 45, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12481, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2261}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12591", 'reported_seq': 12712, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:45.199773+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:45.199773+0000', 'last_peered': '2024-09-11T05:07:45.199773+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:45.199773+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 503536, 'num_objects': 62, 'num_object_clones': 0, 'num_object_copies': 248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 62, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2236}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12523", 'reported_seq': 12635, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:45.194694+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:45.194694+0000', 'last_peered': '2024-09-11T05:07:45.194694+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:45.194694+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 562336, 'num_objects': 69, 'num_object_clones': 0, 'num_object_copies': 276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 69, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12521, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2195}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:49.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:49.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12344", 'reported_seq': 12468, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:44.964392+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:44.964392+0000', 'last_peered': '2024-09-11T05:07:44.964392+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:44.964392+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 529568, 'num_objects': 65, 'num_object_clones': 0, 'num_object_copies': 260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 65, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12342, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:49.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:49.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12738", 'reported_seq': 12863, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:47.331336+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:47.331336+0000', 'last_peered': '2024-09-11T05:07:47.331336+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:47.331336+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 316576, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12736, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2249}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:49.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:50.150 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:50.150 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29020/3380 objects degraded (858.580%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":797,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":845,"data_bytes":6879128,"bytes_used":2806927360,"bytes_avail":383740129280,"bytes_total":386547056640,"degraded_objects":29020,"degraded_total":3380,"degraded_ratio":8.5857988165680474,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:50.172 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:50.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:50.484 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Removed 100649 objects 2024-09-11T05:07:50.484 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi031.stdout:Clean up completed and total clean up time :485.752 2024-09-11T05:07:50.668 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:50.681 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-11T05:07:49.929271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6879128,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173205,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741140,"kb_used_data":16056,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374746220,"statfs":{"total":386547056640,"available":383740129280,"internally_reserved":0,"allocated":16441344,"data_stored":15647308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2790377014},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,4,10,16,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15622144,"num_objects":-1907,"num_object_clones":0,"num_object_copies":-7628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1907,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003061"},"pg_stats":[{"pgid":"1.f","version":"37'12446","reported_seq":12546,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.276977+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:45.276977+0000","last_peered":"2024-09-11T05:07:45.276977+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:45.276977+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":557056,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12446,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12706","reported_seq":12824,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.275083+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:47.275083+0000","last_peered":"2024-09-11T05:07:47.275083+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:47.275083+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259232,"num_objects":32,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12704,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12104","reported_seq":12212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.036478+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:45.036478+0000","last_peered":"2024-09-11T05:07:45.036478+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:45.036478+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457320,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12103,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12365","reported_seq":13822,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.301570+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:47.301570+0000","last_peered":"2024-09-11T05:07:47.301570+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:47.301570+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414880,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12466","reported_seq":12594,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.257155+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:45.257155+0000","last_peered":"2024-09-11T05:07:45.257155+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:45.257155+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529568,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12464,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12580","reported_seq":12691,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.214878+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:45.214878+0000","last_peered":"2024-09-11T05:07:45.214878+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:45.214878+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12691","reported_seq":12790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.153966+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:47.153966+0000","last_peered":"2024-09-11T05:07:47.153966+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:47.153966+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316576,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12689,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12786","reported_seq":17649,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:46.883166+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:46.883166+0000","last_peered":"2024-09-11T05:07:46.883166+0000","last_clean":"2024-09-11T05:07:46.883166+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:46.883166+0000","last_undegraded":"2024-09-11T05:07:46.883166+0000","last_fullsized":"2024-09-11T05:07:46.883166+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303104,"num_objects":37,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12715","reported_seq":12843,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.211719+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:45.211719+0000","last_peered":"2024-09-11T05:07:45.211719+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:45.211719+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":592192,"num_objects":73,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12712,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12483","reported_seq":12599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.058166+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:45.058166+0000","last_peered":"2024-09-11T05:07:45.058166+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:45.058166+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365728,"num_objects":45,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12591","reported_seq":12712,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.199773+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:45.199773+0000","last_peered":"2024-09-11T05:07:45.199773+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:45.199773+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503536,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12358","reported_seq":16964,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:47.026666+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:47.026666+0000","last_peered":"2024-09-11T05:07:47.026666+0000","last_clean":"2024-09-11T05:07:47.026666+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:47.026666+0000","last_undegraded":"2024-09-11T05:07:47.026666+0000","last_fullsized":"2024-09-11T05:07:47.026666+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212992,"num_objects":26,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6168,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12665","reported_seq":17446,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:47.003613+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:47.003613+0000","last_peered":"2024-09-11T05:07:47.003613+0000","last_clean":"2024-09-11T05:07:47.003613+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:47.003613+0000","last_undegraded":"2024-09-11T05:07:47.003613+0000","last_fullsized":"2024-09-11T05:07:47.003613+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":812,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12523","reported_seq":12635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:45.194694+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:45.194694+0000","last_peered":"2024-09-11T05:07:45.194694+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:45.194694+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562336,"num_objects":69,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2195}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12344","reported_seq":12468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:44.964392+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:44.964392+0000","last_peered":"2024-09-11T05:07:44.964392+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:44.964392+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529568,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12342,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12738","reported_seq":12863,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:47.331336+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:47.331336+0000","last_peered":"2024-09-11T05:07:47.331336+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:47.331336+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316576,"num_objects":39,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12736,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6879128,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173205,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13430784,"data_stored":13430784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726116,"kb_used_data":4412,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93645724,"statfs":{"total":96636764160,"available":95893221376,"internally_reserved":0,"allocated":4517888,"data_stored":4319379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":738979536},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722180,"kb_used_data":5172,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93649660,"statfs":{"total":96636764160,"available":95897251840,"internally_reserved":0,"allocated":5296128,"data_stored":5097619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,8,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572156,"kb_used_data":3240,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93799684,"statfs":{"total":96636764160,"available":96050876416,"internally_reserved":0,"allocated":3317760,"data_stored":3119251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":582545106},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720688,"kb_used_data":3232,"kb_used_omap":4,"kb_used_meta":717435,"kb_avail":93651152,"statfs":{"total":96636764160,"available":95898779648,"internally_reserved":0,"allocated":3309568,"data_stored":3111059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":734654159},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2342912,"data_stored":2342912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":2703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4300800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4083712,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:53.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:54.019 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:54.019 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:54.032 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-11T05:07:53.930100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317336,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174006,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762368,"kb_used_data":2624,"kb_used_omap":14,"kb_used_meta":2759665,"kb_avail":374724992,"statfs":{"total":386547056640,"available":383718391808,"internally_reserved":0,"allocated":2686976,"data_stored":1892940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2825897526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,5,3,12,14,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16449536,"num_objects":-2008,"num_object_clones":0,"num_object_copies":-8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002943"},"pg_stats":[{"pgid":"1.f","version":"37'12512","reported_seq":12612,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.186803+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:50.186803+0000","last_peered":"2024-09-11T05:07:50.186803+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:50.186803+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12512,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.434885+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:50.434885+0000","last_peered":"2024-09-11T05:07:50.434885+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:50.434885+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12156","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.881943+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:49.881943+0000","last_peered":"2024-09-11T05:07:49.881943+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:49.881943+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31336,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12155,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":14595,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:52.405716+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:52.405716+0000","last_peered":"2024-09-11T05:07:52.405716+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:52.405716+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":12658,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.266759+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:50.266759+0000","last_peered":"2024-09-11T05:07:50.266759+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:50.266759+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12639","reported_seq":12749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.230941+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:50.230941+0000","last_peered":"2024-09-11T05:07:50.230941+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:50.230941+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8192,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12827,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.311788+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:50.311788+0000","last_peered":"2024-09-11T05:07:50.311788+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:50.311788+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17686,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.378134+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:50.378134+0000","last_peered":"2024-09-11T05:07:50.378134+0000","last_clean":"2024-09-11T05:07:50.378134+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:50.378134+0000","last_undegraded":"2024-09-11T05:07:50.378134+0000","last_fullsized":"2024-09-11T05:07:50.378134+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12782","reported_seq":12910,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.253612+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:50.253612+0000","last_peered":"2024-09-11T05:07:50.253612+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:50.253612+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43328,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12779,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12523","reported_seq":12641,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.775255+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:49.775255+0000","last_peered":"2024-09-11T05:07:49.775255+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:49.775255+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38048,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12648","reported_seq":12770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.045756+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:50.045756+0000","last_peered":"2024-09-11T05:07:50.045756+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:50.045756+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36592,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.420804+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:50.420804+0000","last_peered":"2024-09-11T05:07:50.420804+0000","last_clean":"2024-09-11T05:07:50.420804+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:50.420804+0000","last_undegraded":"2024-09-11T05:07:50.420804+0000","last_fullsized":"2024-09-11T05:07:50.420804+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17504,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.441560+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:50.441560+0000","last_peered":"2024-09-11T05:07:50.441560+0000","last_clean":"2024-09-11T05:07:50.441560+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:50.441560+0000","last_undegraded":"2024-09-11T05:07:50.441560+0000","last_fullsized":"2024-09-11T05:07:50.441560+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12589","reported_seq":12700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.320111+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:50.320111+0000","last_peered":"2024-09-11T05:07:50.320111+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.320111+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21664,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12587,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12402","reported_seq":12526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.974948+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:49.974948+0000","last_peered":"2024-09-11T05:07:49.974948+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:49.974948+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54432,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12400,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.268434+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:50.268434+0000","last_peered":"2024-09-11T05:07:50.268434+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:50.268434+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":317336,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174006,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":589824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744520,"kb_used_data":632,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627320,"statfs":{"total":96636764160,"available":95874375680,"internally_reserved":0,"allocated":647168,"data_stored":448659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,0,6,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722460,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93649380,"statfs":{"total":96636764160,"available":95896965120,"internally_reserved":0,"allocated":1323008,"data_stored":1124499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573464,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798376,"statfs":{"total":96636764160,"available":96049537024,"internally_reserved":0,"allocated":401408,"data_stored":202899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586804946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":90112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:54.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:54.344 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:54.345 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:54.358 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-11T05:07:53.930100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317336,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174006,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762368,"kb_used_data":2624,"kb_used_omap":14,"kb_used_meta":2759665,"kb_avail":374724992,"statfs":{"total":386547056640,"available":383718391808,"internally_reserved":0,"allocated":2686976,"data_stored":1892940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2825897526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,5,3,12,14,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16449536,"num_objects":-2008,"num_object_clones":0,"num_object_copies":-8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002943"},"pg_stats":[{"pgid":"1.f","version":"37'12512","reported_seq":12612,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.186803+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:50.186803+0000","last_peered":"2024-09-11T05:07:50.186803+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:50.186803+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12512,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.434885+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:50.434885+0000","last_peered":"2024-09-11T05:07:50.434885+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:50.434885+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12156","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.881943+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:49.881943+0000","last_peered":"2024-09-11T05:07:49.881943+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:49.881943+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31336,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12155,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":14595,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:52.405716+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:52.405716+0000","last_peered":"2024-09-11T05:07:52.405716+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:52.405716+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":12658,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.266759+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:50.266759+0000","last_peered":"2024-09-11T05:07:50.266759+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:50.266759+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12639","reported_seq":12749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.230941+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:50.230941+0000","last_peered":"2024-09-11T05:07:50.230941+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:50.230941+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8192,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12827,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.311788+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:50.311788+0000","last_peered":"2024-09-11T05:07:50.311788+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:50.311788+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17686,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.378134+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:50.378134+0000","last_peered":"2024-09-11T05:07:50.378134+0000","last_clean":"2024-09-11T05:07:50.378134+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:50.378134+0000","last_undegraded":"2024-09-11T05:07:50.378134+0000","last_fullsized":"2024-09-11T05:07:50.378134+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12782","reported_seq":12910,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.253612+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:50.253612+0000","last_peered":"2024-09-11T05:07:50.253612+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:50.253612+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43328,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12779,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12523","reported_seq":12641,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.775255+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:49.775255+0000","last_peered":"2024-09-11T05:07:49.775255+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:49.775255+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38048,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12648","reported_seq":12770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.045756+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:50.045756+0000","last_peered":"2024-09-11T05:07:50.045756+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:50.045756+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36592,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.420804+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:50.420804+0000","last_peered":"2024-09-11T05:07:50.420804+0000","last_clean":"2024-09-11T05:07:50.420804+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:50.420804+0000","last_undegraded":"2024-09-11T05:07:50.420804+0000","last_fullsized":"2024-09-11T05:07:50.420804+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17504,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.441560+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:50.441560+0000","last_peered":"2024-09-11T05:07:50.441560+0000","last_clean":"2024-09-11T05:07:50.441560+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:50.441560+0000","last_undegraded":"2024-09-11T05:07:50.441560+0000","last_fullsized":"2024-09-11T05:07:50.441560+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12589","reported_seq":12700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.320111+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:50.320111+0000","last_peered":"2024-09-11T05:07:50.320111+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.320111+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21664,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12587,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12402","reported_seq":12526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.974948+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:49.974948+0000","last_peered":"2024-09-11T05:07:49.974948+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:49.974948+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54432,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12400,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.268434+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:50.268434+0000","last_peered":"2024-09-11T05:07:50.268434+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:50.268434+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":317336,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174006,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":589824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744520,"kb_used_data":632,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627320,"statfs":{"total":96636764160,"available":95874375680,"internally_reserved":0,"allocated":647168,"data_stored":448659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,0,6,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722460,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93649380,"statfs":{"total":96636764160,"available":95896965120,"internally_reserved":0,"allocated":1323008,"data_stored":1124499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573464,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798376,"statfs":{"total":96636764160,"available":96049537024,"internally_reserved":0,"allocated":401408,"data_stored":202899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586804946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":90112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:54.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12512", 'reported_seq': 12612, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.186803+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:50.186803+0000', 'last_peered': '2024-09-11T05:07:50.186803+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:50.186803+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16384, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12512, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2289}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12736", 'reported_seq': 12856, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.434885+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:50.434885+0000', 'last_peered': '2024-09-11T05:07:50.434885+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:50.434885+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12156", 'reported_seq': 12266, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:49.881943+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:49.881943+0000', 'last_peered': '2024-09-11T05:07:49.881943+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:49.881943+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31336, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 20, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12155, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2206}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12414", 'reported_seq': 14595, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:52.405716+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:52.405716+0000', 'last_peered': '2024-09-11T05:07:52.405716+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:52.405716+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12412, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1052, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1228}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:54.360 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12529", 'reported_seq': 12658, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.266759+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:50.266759+0000', 'last_peered': '2024-09-11T05:07:50.266759+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:50.266759+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12527, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12639", 'reported_seq': 12749, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.230941+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:50.230941+0000', 'last_peered': '2024-09-11T05:07:50.230941+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:50.230941+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8192, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12639, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2293}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12827, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.311788+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:50.311788+0000', 'last_peered': '2024-09-11T05:07:50.311788+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:50.311788+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12782", 'reported_seq': 12910, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.253612+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:50.253612+0000', 'last_peered': '2024-09-11T05:07:50.253612+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:50.253612+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43328, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12779, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2360}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12523", 'reported_seq': 12641, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:49.775255+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:49.775255+0000', 'last_peered': '2024-09-11T05:07:49.775255+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:49.775255+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38048, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 20, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12521, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12648", 'reported_seq': 12770, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.045756+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:50.045756+0000', 'last_peered': '2024-09-11T05:07:50.045756+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:50.045756+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36592, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 20, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12645, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2293}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:54.361 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12589", 'reported_seq': 12700, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.320111+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:50.320111+0000', 'last_peered': '2024-09-11T05:07:50.320111+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:50.320111+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21664, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12587, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2261}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:54.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:54.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12402", 'reported_seq': 12526, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:49.974948+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:49.974948+0000', 'last_peered': '2024-09-11T05:07:49.974948+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:49.974948+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 54432, 'num_objects': 7, 'num_object_clones': 0, 'num_object_copies': 28, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12400, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:54.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:54.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12900, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.268434+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:50.268434+0000', 'last_peered': '2024-09-11T05:07:50.268434+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:50.268434+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:54.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:54.745 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28658/176 objects degraded (16282.955%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":801,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44,"data_bytes":317336,"bytes_used":2828664832,"bytes_avail":383718391808,"bytes_total":386547056640,"degraded_objects":28658,"degraded_total":176,"degraded_ratio":162.82954545454547,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":167},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:54.758 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:54.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:55.089 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:55.103 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-11T05:07:53.930100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317336,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174006,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762368,"kb_used_data":2624,"kb_used_omap":14,"kb_used_meta":2759665,"kb_avail":374724992,"statfs":{"total":386547056640,"available":383718391808,"internally_reserved":0,"allocated":2686976,"data_stored":1892940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2825897526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,5,3,12,14,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16449536,"num_objects":-2008,"num_object_clones":0,"num_object_copies":-8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002943"},"pg_stats":[{"pgid":"1.f","version":"37'12512","reported_seq":12612,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.186803+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:50.186803+0000","last_peered":"2024-09-11T05:07:50.186803+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:50.186803+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12512,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.434885+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:50.434885+0000","last_peered":"2024-09-11T05:07:50.434885+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:50.434885+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12156","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.881943+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:49.881943+0000","last_peered":"2024-09-11T05:07:49.881943+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:49.881943+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31336,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12155,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":14595,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:52.405716+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:52.405716+0000","last_peered":"2024-09-11T05:07:52.405716+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:52.405716+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":12658,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.266759+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:50.266759+0000","last_peered":"2024-09-11T05:07:50.266759+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:50.266759+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12639","reported_seq":12749,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.230941+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:50.230941+0000","last_peered":"2024-09-11T05:07:50.230941+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:50.230941+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8192,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12827,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.311788+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:50.311788+0000","last_peered":"2024-09-11T05:07:50.311788+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:50.311788+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17686,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.378134+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:50.378134+0000","last_peered":"2024-09-11T05:07:50.378134+0000","last_clean":"2024-09-11T05:07:50.378134+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:50.378134+0000","last_undegraded":"2024-09-11T05:07:50.378134+0000","last_fullsized":"2024-09-11T05:07:50.378134+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12782","reported_seq":12910,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.253612+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:50.253612+0000","last_peered":"2024-09-11T05:07:50.253612+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:50.253612+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43328,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12779,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12523","reported_seq":12641,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.775255+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:49.775255+0000","last_peered":"2024-09-11T05:07:49.775255+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:49.775255+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38048,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12648","reported_seq":12770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.045756+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:50.045756+0000","last_peered":"2024-09-11T05:07:50.045756+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:50.045756+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36592,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.420804+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:50.420804+0000","last_peered":"2024-09-11T05:07:50.420804+0000","last_clean":"2024-09-11T05:07:50.420804+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:50.420804+0000","last_undegraded":"2024-09-11T05:07:50.420804+0000","last_fullsized":"2024-09-11T05:07:50.420804+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17504,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.441560+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:50.441560+0000","last_peered":"2024-09-11T05:07:50.441560+0000","last_clean":"2024-09-11T05:07:50.441560+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:50.441560+0000","last_undegraded":"2024-09-11T05:07:50.441560+0000","last_fullsized":"2024-09-11T05:07:50.441560+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12589","reported_seq":12700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.320111+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:50.320111+0000","last_peered":"2024-09-11T05:07:50.320111+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.320111+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21664,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12587,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12402","reported_seq":12526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:49.974948+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:49.974948+0000","last_peered":"2024-09-11T05:07:49.974948+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:49.974948+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54432,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12400,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.268434+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:50.268434+0000","last_peered":"2024-09-11T05:07:50.268434+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:50.268434+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":317336,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174006,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":589824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744520,"kb_used_data":632,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627320,"statfs":{"total":96636764160,"available":95874375680,"internally_reserved":0,"allocated":647168,"data_stored":448659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,0,6,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722460,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93649380,"statfs":{"total":96636764160,"available":95896965120,"internally_reserved":0,"allocated":1323008,"data_stored":1124499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573464,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798376,"statfs":{"total":96636764160,"available":96049537024,"internally_reserved":0,"allocated":401408,"data_stored":202899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":586804946},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":90112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":360448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:58.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-11T05:07:57.930947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769380,"kb_used_data":1316,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717980,"statfs":{"total":386547056640,"available":383711211520,"internally_reserved":0,"allocated":1347584,"data_stored":553548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6717464,"num_objects":-821,"num_object_clones":0,"num_object_copies":-3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":821,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002172"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.415894+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:50.415894+0000","last_peered":"2024-09-11T05:07:50.415894+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:50.415894+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.434885+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:50.434885+0000","last_peered":"2024-09-11T05:07:50.434885+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:50.434885+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12270,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.451926+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:50.451926+0000","last_peered":"2024-09-11T05:07:50.451926+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:50.451926+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":16737,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:57.406911+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:57.406911+0000","last_peered":"2024-09-11T05:07:57.406911+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:57.406911+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":12658,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.266759+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:50.266759+0000","last_peered":"2024-09-11T05:07:50.266759+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:50.266759+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.401218+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:50.401218+0000","last_peered":"2024-09-11T05:07:50.401218+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:50.401218+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.424051+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:50.424051+0000","last_peered":"2024-09-11T05:07:50.424051+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:50.424051+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17686,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.378134+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:50.378134+0000","last_peered":"2024-09-11T05:07:50.378134+0000","last_clean":"2024-09-11T05:07:50.378134+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:50.378134+0000","last_undegraded":"2024-09-11T05:07:50.378134+0000","last_fullsized":"2024-09-11T05:07:50.378134+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.447647+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:50.447647+0000","last_peered":"2024-09-11T05:07:50.447647+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:50.447647+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.304715+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:50.304715+0000","last_peered":"2024-09-11T05:07:50.304715+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:50.304715+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.419521+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:50.419521+0000","last_peered":"2024-09-11T05:07:50.419521+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:50.419521+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.420804+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:50.420804+0000","last_peered":"2024-09-11T05:07:50.420804+0000","last_clean":"2024-09-11T05:07:50.420804+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:50.420804+0000","last_undegraded":"2024-09-11T05:07:50.420804+0000","last_fullsized":"2024-09-11T05:07:50.420804+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17504,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.441560+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:50.441560+0000","last_peered":"2024-09-11T05:07:50.441560+0000","last_clean":"2024-09-11T05:07:50.441560+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:50.441560+0000","last_undegraded":"2024-09-11T05:07:50.441560+0000","last_fullsized":"2024-09-11T05:07:50.441560+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.399710+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:50.399710+0000","last_peered":"2024-09-11T05:07:50.399710+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.399710+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12531,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.457523+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:50.457523+0000","last_peered":"2024-09-11T05:07:50.457523+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:50.457523+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.268434+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:50.268434+0000","last_peered":"2024-09-11T05:07:50.268434+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:50.268434+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":102400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744196,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627644,"statfs":{"total":96636764160,"available":95874707456,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721476,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650364,"statfs":{"total":96636764160,"available":95897972736,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581784,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790056,"statfs":{"total":96636764160,"available":96041017344,"internally_reserved":0,"allocated":401408,"data_stored":202899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":90112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:58.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:58.878 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:58.890 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-11T05:07:57.930947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769380,"kb_used_data":1316,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717980,"statfs":{"total":386547056640,"available":383711211520,"internally_reserved":0,"allocated":1347584,"data_stored":553548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6717464,"num_objects":-821,"num_object_clones":0,"num_object_copies":-3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":821,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002172"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.415894+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:50.415894+0000","last_peered":"2024-09-11T05:07:50.415894+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:50.415894+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.434885+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:50.434885+0000","last_peered":"2024-09-11T05:07:50.434885+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:50.434885+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12270,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.451926+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:50.451926+0000","last_peered":"2024-09-11T05:07:50.451926+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:50.451926+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":16737,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:57.406911+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:57.406911+0000","last_peered":"2024-09-11T05:07:57.406911+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:57.406911+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":12658,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.266759+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:50.266759+0000","last_peered":"2024-09-11T05:07:50.266759+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:50.266759+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.401218+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:50.401218+0000","last_peered":"2024-09-11T05:07:50.401218+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:50.401218+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.424051+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:50.424051+0000","last_peered":"2024-09-11T05:07:50.424051+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:50.424051+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17686,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.378134+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:50.378134+0000","last_peered":"2024-09-11T05:07:50.378134+0000","last_clean":"2024-09-11T05:07:50.378134+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:50.378134+0000","last_undegraded":"2024-09-11T05:07:50.378134+0000","last_fullsized":"2024-09-11T05:07:50.378134+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.447647+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:50.447647+0000","last_peered":"2024-09-11T05:07:50.447647+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:50.447647+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.304715+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:50.304715+0000","last_peered":"2024-09-11T05:07:50.304715+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:50.304715+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.419521+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:50.419521+0000","last_peered":"2024-09-11T05:07:50.419521+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:50.419521+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.420804+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:50.420804+0000","last_peered":"2024-09-11T05:07:50.420804+0000","last_clean":"2024-09-11T05:07:50.420804+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:50.420804+0000","last_undegraded":"2024-09-11T05:07:50.420804+0000","last_fullsized":"2024-09-11T05:07:50.420804+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17504,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.441560+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:50.441560+0000","last_peered":"2024-09-11T05:07:50.441560+0000","last_clean":"2024-09-11T05:07:50.441560+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:50.441560+0000","last_undegraded":"2024-09-11T05:07:50.441560+0000","last_fullsized":"2024-09-11T05:07:50.441560+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.399710+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:50.399710+0000","last_peered":"2024-09-11T05:07:50.399710+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.399710+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12531,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.457523+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:50.457523+0000","last_peered":"2024-09-11T05:07:50.457523+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:50.457523+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.268434+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:50.268434+0000","last_peered":"2024-09-11T05:07:50.268434+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:50.268434+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":102400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744196,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627644,"statfs":{"total":96636764160,"available":95874707456,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721476,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650364,"statfs":{"total":96636764160,"available":95897972736,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581784,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790056,"statfs":{"total":96636764160,"available":96041017344,"internally_reserved":0,"allocated":401408,"data_stored":202899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":90112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:58.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:07:58.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12614, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.415894+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:50.415894+0000', 'last_peered': '2024-09-11T05:07:50.415894+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:50.415894+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12736", 'reported_seq': 12856, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.434885+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:50.434885+0000', 'last_peered': '2024-09-11T05:07:50.434885+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:50.434885+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12270, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.451926+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:50.451926+0000', 'last_peered': '2024-09-11T05:07:50.451926+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:50.451926+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12414", 'reported_seq': 16737, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:57.406911+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:57.406911+0000', 'last_peered': '2024-09-11T05:07:57.406911+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:57.406911+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 159, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12412, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2123, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12529", 'reported_seq': 12658, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.266759+0000', 'last_change': '2024-09-11T04:56:20.593561+0000', 'last_active': '2024-09-11T05:07:50.266759+0000', 'last_peered': '2024-09-11T05:07:50.266759+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:07:50.266759+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12527, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:07:58.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12751, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.401218+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:50.401218+0000', 'last_peered': '2024-09-11T05:07:50.401218+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:50.401218+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12828, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.424051+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:50.424051+0000', 'last_peered': '2024-09-11T05:07:50.424051+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:50.424051+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12912, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.447647+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:50.447647+0000', 'last_peered': '2024-09-11T05:07:50.447647+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:50.447647+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 12644, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.304715+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:50.304715+0000', 'last_peered': '2024-09-11T05:07:50.304715+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:50.304715+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2304}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12772, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.419521+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:50.419521+0000', 'last_peered': '2024-09-11T05:07:50.419521+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:50.419521+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:07:58.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12702, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.399710+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:50.399710+0000', 'last_peered': '2024-09-11T05:07:50.399710+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:50.399710+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:07:58.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:07:58.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12531, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.457523+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:50.457523+0000', 'last_peered': '2024-09-11T05:07:50.457523+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:50.457523+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:07:58.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:07:58.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12900, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.268434+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:50.268434+0000', 'last_peered': '2024-09-11T05:07:50.268434+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:50.268434+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:07:58.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:07:59.347 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:59.347 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27587/96 objects degraded (28736.458%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":806,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24,"data_bytes":161664,"bytes_used":2835845120,"bytes_avail":383711211520,"bytes_total":386547056640,"degraded_objects":27587,"degraded_total":96,"degraded_ratio":287.36458333333331,"recovering_objects_per_sec":143,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":82},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:07:59.361 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:07:59.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:59.698 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:07:59.699 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:07:59.711 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-11T05:07:58.913255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769380,"kb_used_data":1316,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717980,"statfs":{"total":386547056640,"available":383711211520,"internally_reserved":0,"allocated":1347584,"data_stored":553548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6717464,"num_objects":-821,"num_object_clones":0,"num_object_copies":-3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":821,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.983984"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12614,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.415894+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:50.415894+0000","last_peered":"2024-09-11T05:07:50.415894+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:50.415894+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.434885+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:50.434885+0000","last_peered":"2024-09-11T05:07:50.434885+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:50.434885+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12270,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.451926+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:50.451926+0000","last_peered":"2024-09-11T05:07:50.451926+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:50.451926+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":16737,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:57.406911+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:57.406911+0000","last_peered":"2024-09-11T05:07:57.406911+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:57.406911+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":12658,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.266759+0000","last_change":"2024-09-11T04:56:20.593561+0000","last_active":"2024-09-11T05:07:50.266759+0000","last_peered":"2024-09-11T05:07:50.266759+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:07:50.266759+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12751,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.401218+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:50.401218+0000","last_peered":"2024-09-11T05:07:50.401218+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:50.401218+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.424051+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:50.424051+0000","last_peered":"2024-09-11T05:07:50.424051+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:50.424051+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17686,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.378134+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:50.378134+0000","last_peered":"2024-09-11T05:07:50.378134+0000","last_clean":"2024-09-11T05:07:50.378134+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:50.378134+0000","last_undegraded":"2024-09-11T05:07:50.378134+0000","last_fullsized":"2024-09-11T05:07:50.378134+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.447647+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:50.447647+0000","last_peered":"2024-09-11T05:07:50.447647+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:50.447647+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12644,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.304715+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:50.304715+0000","last_peered":"2024-09-11T05:07:50.304715+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:50.304715+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.419521+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:50.419521+0000","last_peered":"2024-09-11T05:07:50.419521+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:50.419521+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.420804+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:50.420804+0000","last_peered":"2024-09-11T05:07:50.420804+0000","last_clean":"2024-09-11T05:07:50.420804+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:50.420804+0000","last_undegraded":"2024-09-11T05:07:50.420804+0000","last_fullsized":"2024-09-11T05:07:50.420804+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17504,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.441560+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:50.441560+0000","last_peered":"2024-09-11T05:07:50.441560+0000","last_clean":"2024-09-11T05:07:50.441560+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:50.441560+0000","last_undegraded":"2024-09-11T05:07:50.441560+0000","last_fullsized":"2024-09-11T05:07:50.441560+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.399710+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:50.399710+0000","last_peered":"2024-09-11T05:07:50.399710+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.399710+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12531,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.457523+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:50.457523+0000","last_peered":"2024-09-11T05:07:50.457523+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:50.457523+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.268434+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:50.268434+0000","last_peered":"2024-09-11T05:07:50.268434+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:50.268434+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":102400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744196,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627644,"statfs":{"total":96636764160,"available":95874707456,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721476,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650364,"statfs":{"total":96636764160,"available":95897972736,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581784,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790056,"statfs":{"total":96636764160,"available":96041017344,"internally_reserved":0,"allocated":401408,"data_stored":202899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":90112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:02.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:03.029 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:03.041 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-11T05:08:01.931750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769428,"kb_used_data":1348,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717932,"statfs":{"total":386547056640,"available":383711162368,"internally_reserved":0,"allocated":1380352,"data_stored":567554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-155672,"num_objects":-20,"num_object_clones":0,"num_object_copies":-80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001649"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12618,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918024+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:58.918024+0000","last_peered":"2024-09-11T05:07:58.918024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:58.918024+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.434885+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:50.434885+0000","last_peered":"2024-09-11T05:07:50.434885+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:50.434885+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.915835+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:58.915835+0000","last_peered":"2024-09-11T05:07:58.915835+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:58.915835+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":16737,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:57.406911+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:57.406911+0000","last_peered":"2024-09-11T05:07:57.406911+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:57.406911+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":13677,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:00.394396+0000","last_change":"2024-09-11T05:07:57.875622+0000","last_active":"2024-09-11T05:08:00.394396+0000","last_peered":"2024-09-11T05:08:00.394396+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:08:00.394396+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12755,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:58.917060+0000","last_peered":"2024-09-11T05:07:58.917060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:58.917060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.424051+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:50.424051+0000","last_peered":"2024-09-11T05:07:50.424051+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:50.424051+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17686,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.378134+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:50.378134+0000","last_peered":"2024-09-11T05:07:50.378134+0000","last_clean":"2024-09-11T05:07:50.378134+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:50.378134+0000","last_undegraded":"2024-09-11T05:07:50.378134+0000","last_fullsized":"2024-09-11T05:07:50.378134+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917005+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:58.917005+0000","last_peered":"2024-09-11T05:07:58.917005+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:58.917005+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.916239+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:58.916239+0000","last_peered":"2024-09-11T05:07:58.916239+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:58.916239+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12776,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917202+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:58.917202+0000","last_peered":"2024-09-11T05:07:58.917202+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:58.917202+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.420804+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:50.420804+0000","last_peered":"2024-09-11T05:07:50.420804+0000","last_clean":"2024-09-11T05:07:50.420804+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:50.420804+0000","last_undegraded":"2024-09-11T05:07:50.420804+0000","last_fullsized":"2024-09-11T05:07:50.420804+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17504,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.441560+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:50.441560+0000","last_peered":"2024-09-11T05:07:50.441560+0000","last_clean":"2024-09-11T05:07:50.441560+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:50.441560+0000","last_undegraded":"2024-09-11T05:07:50.441560+0000","last_fullsized":"2024-09-11T05:07:50.441560+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918173+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:58.918173+0000","last_peered":"2024-09-11T05:07:58.918173+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:58.918173+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.914784+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:58.914784+0000","last_peered":"2024-09-11T05:07:58.914784+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:58.914784+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.268434+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:50.268434+0000","last_peered":"2024-09-11T05:07:50.268434+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:50.268434+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":102400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744220,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627620,"statfs":{"total":96636764160,"available":95874682880,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581784,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790056,"statfs":{"total":96636764160,"available":96041017344,"internally_reserved":0,"allocated":401408,"data_stored":202899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":90112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:03.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:03.390 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-11T05:08:01.931750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769428,"kb_used_data":1348,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717932,"statfs":{"total":386547056640,"available":383711162368,"internally_reserved":0,"allocated":1380352,"data_stored":567554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-155672,"num_objects":-20,"num_object_clones":0,"num_object_copies":-80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001649"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12618,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918024+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:58.918024+0000","last_peered":"2024-09-11T05:07:58.918024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:58.918024+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.434885+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:50.434885+0000","last_peered":"2024-09-11T05:07:50.434885+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:50.434885+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.915835+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:58.915835+0000","last_peered":"2024-09-11T05:07:58.915835+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:58.915835+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":16737,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:57.406911+0000","last_change":"2024-09-11T05:06:36.732753+0000","last_active":"2024-09-11T05:07:57.406911+0000","last_peered":"2024-09-11T05:07:57.406911+0000","last_clean":"2024-09-11T04:55:08.102450+0000","last_became_active":"2024-09-11T04:56:20.565392+0000","last_became_peered":"2024-09-11T04:56:20.565392+0000","last_unstale":"2024-09-11T05:07:57.406911+0000","last_undegraded":"2024-09-11T04:56:20.289747+0000","last_fullsized":"2024-09-11T04:56:20.222983+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":13677,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:00.394396+0000","last_change":"2024-09-11T05:07:57.875622+0000","last_active":"2024-09-11T05:08:00.394396+0000","last_peered":"2024-09-11T05:08:00.394396+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:08:00.394396+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12755,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:58.917060+0000","last_peered":"2024-09-11T05:07:58.917060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:58.917060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.424051+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:50.424051+0000","last_peered":"2024-09-11T05:07:50.424051+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:50.424051+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17686,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.378134+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:50.378134+0000","last_peered":"2024-09-11T05:07:50.378134+0000","last_clean":"2024-09-11T05:07:50.378134+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:50.378134+0000","last_undegraded":"2024-09-11T05:07:50.378134+0000","last_fullsized":"2024-09-11T05:07:50.378134+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917005+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:58.917005+0000","last_peered":"2024-09-11T05:07:58.917005+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:58.917005+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.916239+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:58.916239+0000","last_peered":"2024-09-11T05:07:58.916239+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:58.916239+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12776,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917202+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:58.917202+0000","last_peered":"2024-09-11T05:07:58.917202+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:58.917202+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.420804+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:50.420804+0000","last_peered":"2024-09-11T05:07:50.420804+0000","last_clean":"2024-09-11T05:07:50.420804+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:50.420804+0000","last_undegraded":"2024-09-11T05:07:50.420804+0000","last_fullsized":"2024-09-11T05:07:50.420804+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17504,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.441560+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:50.441560+0000","last_peered":"2024-09-11T05:07:50.441560+0000","last_clean":"2024-09-11T05:07:50.441560+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:50.441560+0000","last_undegraded":"2024-09-11T05:07:50.441560+0000","last_fullsized":"2024-09-11T05:07:50.441560+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918173+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:58.918173+0000","last_peered":"2024-09-11T05:07:58.918173+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:58.918173+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.914784+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:58.914784+0000","last_peered":"2024-09-11T05:07:58.914784+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:58.914784+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:50.268434+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:50.268434+0000","last_peered":"2024-09-11T05:07:50.268434+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:50.268434+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":161664,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174026,"num_write_kb":653825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":102400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744220,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627620,"statfs":{"total":96636764160,"available":95874682880,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581784,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790056,"statfs":{"total":96636764160,"available":96041017344,"internally_reserved":0,"allocated":401408,"data_stored":202899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":90112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:03.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:03.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12618, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.918024+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:58.918024+0000', 'last_peered': '2024-09-11T05:07:58.918024+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:58.918024+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:03.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:08:03.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12736", 'reported_seq': 12856, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.434885+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:50.434885+0000', 'last_peered': '2024-09-11T05:07:50.434885+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:50.434885+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:03.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:03.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12274, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.915835+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:58.915835+0000', 'last_peered': '2024-09-11T05:07:58.915835+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:58.915835+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:03.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-11T05:08:03.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12414", 'reported_seq': 16737, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:57.406911+0000', 'last_change': '2024-09-11T05:06:36.732753+0000', 'last_active': '2024-09-11T05:07:57.406911+0000', 'last_peered': '2024-09-11T05:07:57.406911+0000', 'last_clean': '2024-09-11T04:55:08.102450+0000', 'last_became_active': '2024-09-11T04:56:20.565392+0000', 'last_became_peered': '2024-09-11T04:56:20.565392+0000', 'last_unstale': '2024-09-11T05:07:57.406911+0000', 'last_undegraded': '2024-09-11T04:56:20.289747+0000', 'last_fullsized': '2024-09-11T04:56:20.222983+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 159, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12412, 'num_write_kb': 49653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2123, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12529", 'reported_seq': 13677, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:00.394396+0000', 'last_change': '2024-09-11T05:07:57.875622+0000', 'last_active': '2024-09-11T05:08:00.394396+0000', 'last_peered': '2024-09-11T05:08:00.394396+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:08:00.394396+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1716, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12527, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 538, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1714}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12755, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.917060+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:58.917060+0000', 'last_peered': '2024-09-11T05:07:58.917060+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:58.917060+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12828, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.424051+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:50.424051+0000', 'last_peered': '2024-09-11T05:07:50.424051+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:50.424051+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12916, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.917005+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:58.917005+0000', 'last_peered': '2024-09-11T05:07:58.917005+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:58.917005+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:08:03.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 12648, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.916239+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:58.916239+0000', 'last_peered': '2024-09-11T05:07:58.916239+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:58.916239+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2304}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:03.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:03.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12776, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.917202+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:58.917202+0000', 'last_peered': '2024-09-11T05:07:58.917202+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:58.917202+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:03.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:03.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12706, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.918173+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:58.918173+0000', 'last_peered': '2024-09-11T05:07:58.918173+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:58.918173+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:03.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:03.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12535, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.914784+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:58.914784+0000', 'last_peered': '2024-09-11T05:07:58.914784+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:58.914784+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:03.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:03.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12900, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:50.268434+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:50.268434+0000', 'last_peered': '2024-09-11T05:07:50.268434+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:50.268434+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:03.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:03.788 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27080/96 objects degraded (28208.333%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":810,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":24,"data_bytes":161664,"bytes_used":2835894272,"bytes_avail":383711162368,"bytes_total":386547056640,"degraded_objects":27080,"degraded_total":96,"degraded_ratio":282.08333333333331,"recovering_objects_per_sec":197,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1578,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-11T05:02:15.771141+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:03.802 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:08:03.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:04.139 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:04.152 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-09-11T05:08:03.932105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":771,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765296,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2763825,"kb_avail":374722064,"statfs":{"total":386547056640,"available":383715393536,"internally_reserved":0,"allocated":1404928,"data_stored":573368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2830157366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12412,"num_write_kb":-49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-545,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001629"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12618,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918024+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:58.918024+0000","last_peered":"2024-09-11T05:07:58.918024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:58.918024+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.923123+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:58.923123+0000","last_peered":"2024-09-11T05:07:58.923123+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:58.923123+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.915835+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:58.915835+0000","last_peered":"2024-09-11T05:07:58.915835+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:58.915835+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:59.779704+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:07:59.779704+0000","last_peered":"2024-09-11T05:07:59.779704+0000","last_clean":"2024-09-11T05:07:59.779704+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:07:59.779704+0000","last_undegraded":"2024-09-11T05:07:59.779704+0000","last_fullsized":"2024-09-11T05:07:59.779704+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":13677,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:00.394396+0000","last_change":"2024-09-11T05:07:57.875622+0000","last_active":"2024-09-11T05:08:00.394396+0000","last_peered":"2024-09-11T05:08:00.394396+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:08:00.394396+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12755,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:58.917060+0000","last_peered":"2024-09-11T05:07:58.917060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:58.917060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922685+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:58.922685+0000","last_peered":"2024-09-11T05:07:58.922685+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:58.922685+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17690,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.918870+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:58.918870+0000","last_peered":"2024-09-11T05:07:58.918870+0000","last_clean":"2024-09-11T05:07:58.918870+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:58.918870+0000","last_undegraded":"2024-09-11T05:07:58.918870+0000","last_fullsized":"2024-09-11T05:07:58.918870+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917005+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:58.917005+0000","last_peered":"2024-09-11T05:07:58.917005+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:58.917005+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.916239+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:58.916239+0000","last_peered":"2024-09-11T05:07:58.916239+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:58.916239+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12776,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917202+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:58.917202+0000","last_peered":"2024-09-11T05:07:58.917202+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:58.917202+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.919501+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:58.919501+0000","last_peered":"2024-09-11T05:07:58.919501+0000","last_clean":"2024-09-11T05:07:58.919501+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:58.919501+0000","last_undegraded":"2024-09-11T05:07:58.919501+0000","last_fullsized":"2024-09-11T05:07:58.919501+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17508,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.920018+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:58.920018+0000","last_peered":"2024-09-11T05:07:58.920018+0000","last_clean":"2024-09-11T05:07:58.920018+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:58.920018+0000","last_undegraded":"2024-09-11T05:07:58.920018+0000","last_fullsized":"2024-09-11T05:07:58.920018+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918173+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:58.918173+0000","last_peered":"2024-09-11T05:07:58.918173+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:58.918173+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.914784+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:58.914784+0000","last_peered":"2024-09-11T05:07:58.914784+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:58.914784+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12904,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922687+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:58.922687+0000","last_peered":"2024-09-11T05:07:58.922687+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:58.922687+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":771,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744220,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627620,"statfs":{"total":96636764160,"available":95874682880,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577632,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794208,"statfs":{"total":96636764160,"available":96045268992,"internally_reserved":0,"allocated":409600,"data_stored":201710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591064786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721944,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649896,"statfs":{"total":96636764160,"available":95897493504,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:07.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:07.497 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:07.498 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:07.512 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-11T05:08:05.932477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765296,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2763825,"kb_avail":374722064,"statfs":{"total":386547056640,"available":383715393536,"internally_reserved":0,"allocated":1404928,"data_stored":573368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2830157366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12412,"num_write_kb":-49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.024396"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12618,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918024+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:58.918024+0000","last_peered":"2024-09-11T05:07:58.918024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:58.918024+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.923123+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:58.923123+0000","last_peered":"2024-09-11T05:07:58.923123+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:58.923123+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.915835+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:58.915835+0000","last_peered":"2024-09-11T05:07:58.915835+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:58.915835+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:59.779704+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:07:59.779704+0000","last_peered":"2024-09-11T05:07:59.779704+0000","last_clean":"2024-09-11T05:07:59.779704+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:07:59.779704+0000","last_undegraded":"2024-09-11T05:07:59.779704+0000","last_fullsized":"2024-09-11T05:07:59.779704+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":15423,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:05.393963+0000","last_change":"2024-09-11T05:07:57.875622+0000","last_active":"2024-09-11T05:08:05.393963+0000","last_peered":"2024-09-11T05:08:05.393963+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:08:05.393963+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12755,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:58.917060+0000","last_peered":"2024-09-11T05:07:58.917060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:58.917060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922685+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:58.922685+0000","last_peered":"2024-09-11T05:07:58.922685+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:58.922685+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17690,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.918870+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:58.918870+0000","last_peered":"2024-09-11T05:07:58.918870+0000","last_clean":"2024-09-11T05:07:58.918870+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:58.918870+0000","last_undegraded":"2024-09-11T05:07:58.918870+0000","last_fullsized":"2024-09-11T05:07:58.918870+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917005+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:58.917005+0000","last_peered":"2024-09-11T05:07:58.917005+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:58.917005+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.916239+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:58.916239+0000","last_peered":"2024-09-11T05:07:58.916239+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:58.916239+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12776,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917202+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:58.917202+0000","last_peered":"2024-09-11T05:07:58.917202+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:58.917202+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.919501+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:58.919501+0000","last_peered":"2024-09-11T05:07:58.919501+0000","last_clean":"2024-09-11T05:07:58.919501+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:58.919501+0000","last_undegraded":"2024-09-11T05:07:58.919501+0000","last_fullsized":"2024-09-11T05:07:58.919501+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17508,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.920018+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:58.920018+0000","last_peered":"2024-09-11T05:07:58.920018+0000","last_clean":"2024-09-11T05:07:58.920018+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:58.920018+0000","last_undegraded":"2024-09-11T05:07:58.920018+0000","last_fullsized":"2024-09-11T05:07:58.920018+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918173+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:58.918173+0000","last_peered":"2024-09-11T05:07:58.918173+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:58.918173+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.914784+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:58.914784+0000","last_peered":"2024-09-11T05:07:58.914784+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:58.914784+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12904,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922687+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:58.922687+0000","last_peered":"2024-09-11T05:07:58.922687+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:58.922687+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744220,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627620,"statfs":{"total":96636764160,"available":95874682880,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577632,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794208,"statfs":{"total":96636764160,"available":96045268992,"internally_reserved":0,"allocated":409600,"data_stored":201710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591064786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721944,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649896,"statfs":{"total":96636764160,"available":95897493504,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:07.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:07.848 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:07.849 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:07.861 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-11T05:08:05.932477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765296,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2763825,"kb_avail":374722064,"statfs":{"total":386547056640,"available":383715393536,"internally_reserved":0,"allocated":1404928,"data_stored":573368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2830157366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12412,"num_write_kb":-49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.024396"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12618,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918024+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:58.918024+0000","last_peered":"2024-09-11T05:07:58.918024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:58.918024+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.923123+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:58.923123+0000","last_peered":"2024-09-11T05:07:58.923123+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:58.923123+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.915835+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:58.915835+0000","last_peered":"2024-09-11T05:07:58.915835+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:58.915835+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:59.779704+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:07:59.779704+0000","last_peered":"2024-09-11T05:07:59.779704+0000","last_clean":"2024-09-11T05:07:59.779704+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:07:59.779704+0000","last_undegraded":"2024-09-11T05:07:59.779704+0000","last_fullsized":"2024-09-11T05:07:59.779704+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":15423,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:05.393963+0000","last_change":"2024-09-11T05:07:57.875622+0000","last_active":"2024-09-11T05:08:05.393963+0000","last_peered":"2024-09-11T05:08:05.393963+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:08:05.393963+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12755,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:58.917060+0000","last_peered":"2024-09-11T05:07:58.917060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:58.917060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922685+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:58.922685+0000","last_peered":"2024-09-11T05:07:58.922685+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:58.922685+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17690,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.918870+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:58.918870+0000","last_peered":"2024-09-11T05:07:58.918870+0000","last_clean":"2024-09-11T05:07:58.918870+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:58.918870+0000","last_undegraded":"2024-09-11T05:07:58.918870+0000","last_fullsized":"2024-09-11T05:07:58.918870+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917005+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:58.917005+0000","last_peered":"2024-09-11T05:07:58.917005+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:58.917005+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.916239+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:58.916239+0000","last_peered":"2024-09-11T05:07:58.916239+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:58.916239+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12776,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917202+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:58.917202+0000","last_peered":"2024-09-11T05:07:58.917202+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:58.917202+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.919501+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:58.919501+0000","last_peered":"2024-09-11T05:07:58.919501+0000","last_clean":"2024-09-11T05:07:58.919501+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:58.919501+0000","last_undegraded":"2024-09-11T05:07:58.919501+0000","last_fullsized":"2024-09-11T05:07:58.919501+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17508,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.920018+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:58.920018+0000","last_peered":"2024-09-11T05:07:58.920018+0000","last_clean":"2024-09-11T05:07:58.920018+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:58.920018+0000","last_undegraded":"2024-09-11T05:07:58.920018+0000","last_fullsized":"2024-09-11T05:07:58.920018+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918173+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:58.918173+0000","last_peered":"2024-09-11T05:07:58.918173+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:58.918173+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.914784+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:58.914784+0000","last_peered":"2024-09-11T05:07:58.914784+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:58.914784+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12904,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922687+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:58.922687+0000","last_peered":"2024-09-11T05:07:58.922687+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:58.922687+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744220,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627620,"statfs":{"total":96636764160,"available":95874682880,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577632,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794208,"statfs":{"total":96636764160,"available":96045268992,"internally_reserved":0,"allocated":409600,"data_stored":201710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591064786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721944,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649896,"statfs":{"total":96636764160,"available":95897493504,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:07.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:07.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12618, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.918024+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:07:58.918024+0000', 'last_peered': '2024-09-11T05:07:58.918024+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:07:58.918024+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:07.862 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:08:07.862 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12736", 'reported_seq': 12860, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.923123+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:58.923123+0000', 'last_peered': '2024-09-11T05:07:58.923123+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:58.923123+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12274, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.915835+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:07:58.915835+0000', 'last_peered': '2024-09-11T05:07:58.915835+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:07:58.915835+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12529", 'reported_seq': 15423, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:05.393963+0000', 'last_change': '2024-09-11T05:07:57.875622+0000', 'last_active': '2024-09-11T05:08:05.393963+0000', 'last_peered': '2024-09-11T05:08:05.393963+0000', 'last_clean': '2024-09-11T04:55:07.678387+0000', 'last_became_active': '2024-09-11T04:56:20.583879+0000', 'last_became_peered': '2024-09-11T04:56:20.583879+0000', 'last_unstale': '2024-09-11T05:08:05.393963+0000', 'last_undegraded': '2024-09-11T04:56:20.309515+0000', 'last_fullsized': '2024-09-11T04:56:20.281862+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 843, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12527, 'num_write_kb': 50117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1411, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -841}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12755, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.917060+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:07:58.917060+0000', 'last_peered': '2024-09-11T05:07:58.917060+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:07:58.917060+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12832, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.922685+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:58.922685+0000', 'last_peered': '2024-09-11T05:07:58.922685+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:58.922685+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12916, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.917005+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:07:58.917005+0000', 'last_peered': '2024-09-11T05:07:58.917005+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:07:58.917005+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:07.863 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 12648, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.916239+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:07:58.916239+0000', 'last_peered': '2024-09-11T05:07:58.916239+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:07:58.916239+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2304}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12776, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.917202+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:07:58.917202+0000', 'last_peered': '2024-09-11T05:07:58.917202+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:07:58.917202+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12706, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.918173+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:07:58.918173+0000', 'last_peered': '2024-09-11T05:07:58.918173+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:58.918173+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12535, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.914784+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:07:58.914784+0000', 'last_peered': '2024-09-11T05:07:58.914784+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:07:58.914784+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:07.864 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12904, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.922687+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:58.922687+0000', 'last_peered': '2024-09-11T05:07:58.922687+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:58.922687+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:07.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:08.247 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:08.248 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26048/88 objects degraded (29600.000%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":815,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22,"data_bytes":148192,"bytes_used":2835922944,"bytes_avail":383711133696,"bytes_total":386547056640,"degraded_objects":26048,"degraded_total":88,"degraded_ratio":296},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:08.261 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:08:08.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:08.600 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:08.601 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:08.613 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-11T05:08:07.932932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769456,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717904,"statfs":{"total":386547056640,"available":383711133696,"internally_reserved":0,"allocated":1404928,"data_stored":573368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12412,"num_write_kb":-49653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001985"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12618,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918024+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:07:58.918024+0000","last_peered":"2024-09-11T05:07:58.918024+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:07:58.918024+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.923123+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:58.923123+0000","last_peered":"2024-09-11T05:07:58.923123+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:58.923123+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12274,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.915835+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:07:58.915835+0000","last_peered":"2024-09-11T05:07:58.915835+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:07:58.915835+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:59.779704+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:07:59.779704+0000","last_peered":"2024-09-11T05:07:59.779704+0000","last_clean":"2024-09-11T05:07:59.779704+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:07:59.779704+0000","last_undegraded":"2024-09-11T05:07:59.779704+0000","last_fullsized":"2024-09-11T05:07:59.779704+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":15423,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:05.393963+0000","last_change":"2024-09-11T05:07:57.875622+0000","last_active":"2024-09-11T05:08:05.393963+0000","last_peered":"2024-09-11T05:08:05.393963+0000","last_clean":"2024-09-11T04:55:07.678387+0000","last_became_active":"2024-09-11T04:56:20.583879+0000","last_became_peered":"2024-09-11T04:56:20.583879+0000","last_unstale":"2024-09-11T05:08:05.393963+0000","last_undegraded":"2024-09-11T04:56:20.309515+0000","last_fullsized":"2024-09-11T04:56:20.281862+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12755,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917060+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:07:58.917060+0000","last_peered":"2024-09-11T05:07:58.917060+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:07:58.917060+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922685+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:58.922685+0000","last_peered":"2024-09-11T05:07:58.922685+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:58.922685+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17690,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.918870+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:58.918870+0000","last_peered":"2024-09-11T05:07:58.918870+0000","last_clean":"2024-09-11T05:07:58.918870+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:58.918870+0000","last_undegraded":"2024-09-11T05:07:58.918870+0000","last_fullsized":"2024-09-11T05:07:58.918870+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917005+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:07:58.917005+0000","last_peered":"2024-09-11T05:07:58.917005+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:07:58.917005+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12648,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.916239+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:07:58.916239+0000","last_peered":"2024-09-11T05:07:58.916239+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:07:58.916239+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12776,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.917202+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:07:58.917202+0000","last_peered":"2024-09-11T05:07:58.917202+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:07:58.917202+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.919501+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:58.919501+0000","last_peered":"2024-09-11T05:07:58.919501+0000","last_clean":"2024-09-11T05:07:58.919501+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:58.919501+0000","last_undegraded":"2024-09-11T05:07:58.919501+0000","last_fullsized":"2024-09-11T05:07:58.919501+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17508,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.920018+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:58.920018+0000","last_peered":"2024-09-11T05:07:58.920018+0000","last_clean":"2024-09-11T05:07:58.920018+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:58.920018+0000","last_undegraded":"2024-09-11T05:07:58.920018+0000","last_fullsized":"2024-09-11T05:07:58.920018+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.918173+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:07:58.918173+0000","last_peered":"2024-09-11T05:07:58.918173+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:58.918173+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12535,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.914784+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:07:58.914784+0000","last_peered":"2024-09-11T05:07:58.914784+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:07:58.914784+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12904,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922687+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:58.922687+0000","last_peered":"2024-09-11T05:07:58.922687+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:58.922687+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744220,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627620,"statfs":{"total":96636764160,"available":95874682880,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":201710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721944,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649896,"statfs":{"total":96636764160,"available":95897493504,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:11.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:11.961 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:11.974 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-11T05:08:11.933698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769468,"kb_used_data":1388,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717892,"statfs":{"total":386547056640,"available":383711121408,"internally_reserved":0,"allocated":1421312,"data_stored":580297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001592"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.923123+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:58.923123+0000","last_peered":"2024-09-11T05:07:58.923123+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:58.923123+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12276,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217538+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:09.217538+0000","last_peered":"2024-09-11T05:08:09.217538+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:09.217538+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:59.779704+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:07:59.779704+0000","last_peered":"2024-09-11T05:07:59.779704+0000","last_clean":"2024-09-11T05:07:59.779704+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:07:59.779704+0000","last_undegraded":"2024-09-11T05:07:59.779704+0000","last_fullsized":"2024-09-11T05:07:59.779704+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17122,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.324998+0000","last_change":"2024-09-11T05:08:10.324876+0000","last_active":"2024-09-11T05:08:10.324998+0000","last_peered":"2024-09-11T05:08:10.324998+0000","last_clean":"2024-09-11T05:08:10.324998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:10.324998+0000","last_undegraded":"2024-09-11T05:08:10.324998+0000","last_fullsized":"2024-09-11T05:08:10.324998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2254,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922685+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:58.922685+0000","last_peered":"2024-09-11T05:07:58.922685+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:58.922685+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17690,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.918870+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:58.918870+0000","last_peered":"2024-09-11T05:07:58.918870+0000","last_clean":"2024-09-11T05:07:58.918870+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:58.918870+0000","last_undegraded":"2024-09-11T05:07:58.918870+0000","last_fullsized":"2024-09-11T05:07:58.918870+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12650,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217446+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:09.217446+0000","last_peered":"2024-09-11T05:08:09.217446+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:09.217446+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.919501+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:58.919501+0000","last_peered":"2024-09-11T05:07:58.919501+0000","last_clean":"2024-09-11T05:07:58.919501+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:58.919501+0000","last_undegraded":"2024-09-11T05:07:58.919501+0000","last_fullsized":"2024-09-11T05:07:58.919501+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17508,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.920018+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:58.920018+0000","last_peered":"2024-09-11T05:07:58.920018+0000","last_clean":"2024-09-11T05:07:58.920018+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:58.920018+0000","last_undegraded":"2024-09-11T05:07:58.920018+0000","last_fullsized":"2024-09-11T05:07:58.920018+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12537,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217076+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:09.217076+0000","last_peered":"2024-09-11T05:08:09.217076+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:09.217076+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12904,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922687+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:58.922687+0000","last_peered":"2024-09-11T05:07:58.922687+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:58.922687+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":201710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721944,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649896,"statfs":{"total":96636764160,"available":95897493504,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:11.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:12.310 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:12.310 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:12.322 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-11T05:08:11.933698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769468,"kb_used_data":1388,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717892,"statfs":{"total":386547056640,"available":383711121408,"internally_reserved":0,"allocated":1421312,"data_stored":580297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001592"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.923123+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:58.923123+0000","last_peered":"2024-09-11T05:07:58.923123+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:58.923123+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12276,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217538+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:09.217538+0000","last_peered":"2024-09-11T05:08:09.217538+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:09.217538+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:59.779704+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:07:59.779704+0000","last_peered":"2024-09-11T05:07:59.779704+0000","last_clean":"2024-09-11T05:07:59.779704+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:07:59.779704+0000","last_undegraded":"2024-09-11T05:07:59.779704+0000","last_fullsized":"2024-09-11T05:07:59.779704+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17122,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.324998+0000","last_change":"2024-09-11T05:08:10.324876+0000","last_active":"2024-09-11T05:08:10.324998+0000","last_peered":"2024-09-11T05:08:10.324998+0000","last_clean":"2024-09-11T05:08:10.324998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:10.324998+0000","last_undegraded":"2024-09-11T05:08:10.324998+0000","last_fullsized":"2024-09-11T05:08:10.324998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2254,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922685+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:58.922685+0000","last_peered":"2024-09-11T05:07:58.922685+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:58.922685+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17690,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.918870+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:58.918870+0000","last_peered":"2024-09-11T05:07:58.918870+0000","last_clean":"2024-09-11T05:07:58.918870+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:58.918870+0000","last_undegraded":"2024-09-11T05:07:58.918870+0000","last_fullsized":"2024-09-11T05:07:58.918870+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12650,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217446+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:09.217446+0000","last_peered":"2024-09-11T05:08:09.217446+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:09.217446+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.919501+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:58.919501+0000","last_peered":"2024-09-11T05:07:58.919501+0000","last_clean":"2024-09-11T05:07:58.919501+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:58.919501+0000","last_undegraded":"2024-09-11T05:07:58.919501+0000","last_fullsized":"2024-09-11T05:07:58.919501+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17508,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.920018+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:58.920018+0000","last_peered":"2024-09-11T05:07:58.920018+0000","last_clean":"2024-09-11T05:07:58.920018+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:58.920018+0000","last_undegraded":"2024-09-11T05:07:58.920018+0000","last_fullsized":"2024-09-11T05:07:58.920018+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12537,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217076+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:09.217076+0000","last_peered":"2024-09-11T05:08:09.217076+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:09.217076+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12904,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922687+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:58.922687+0000","last_peered":"2024-09-11T05:07:58.922687+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:58.922687+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":201710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721944,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649896,"statfs":{"total":96636764160,"available":95897493504,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:12.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:12.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12622, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.224830+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:10.224830+0000', 'last_peered': '2024-09-11T05:08:10.224830+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:10.224830+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:12.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:08:12.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12736", 'reported_seq': 12860, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.923123+0000', 'last_change': '2024-09-11T04:56:20.598581+0000', 'last_active': '2024-09-11T05:07:58.923123+0000', 'last_peered': '2024-09-11T05:07:58.923123+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:07:58.923123+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2260, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:12.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:12.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12276, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:09.217538+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:09.217538+0000', 'last_peered': '2024-09-11T05:08:09.217538+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:09.217538+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:12.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:12.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12759, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.222630+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:10.222630+0000', 'last_peered': '2024-09-11T05:08:10.222630+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:10.222630+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12832, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.922685+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:07:58.922685+0000', 'last_peered': '2024-09-11T05:07:58.922685+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:07:58.922685+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12921, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.223741+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:10.223741+0000', 'last_peered': '2024-09-11T05:08:10.223741+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:10.223741+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 12650, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:09.217446+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:08:09.217446+0000', 'last_peered': '2024-09-11T05:08:09.217446+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:08:09.217446+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2304}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12781, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.225785+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:10.225785+0000', 'last_peered': '2024-09-11T05:08:10.225785+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:10.225785+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:12.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:12.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12710, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.222777+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:08:10.222777+0000', 'last_peered': '2024-09-11T05:08:10.222777+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:10.222777+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:12.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:12.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12537, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:09.217076+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:09.217076+0000', 'last_peered': '2024-09-11T05:08:09.217076+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:09.217076+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:12.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:12.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12904, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:07:58.922687+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:07:58.922687+0000', 'last_peered': '2024-09-11T05:07:58.922687+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:07:58.922687+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:12.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:12.707 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:12.707 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25205/88 objects degraded (28642.045%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":819,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":5}],"num_pgs":16,"num_pools":1,"num_objects":22,"data_bytes":148192,"bytes_used":2835935232,"bytes_avail":383711121408,"bytes_total":386547056640,"degraded_objects":25205,"degraded_total":88,"degraded_ratio":286.42045454545456,"recovering_objects_per_sec":214,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1716,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:12.720 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:08:12.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:13.097 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:13.110 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-11T05:08:11.933698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769468,"kb_used_data":1388,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717892,"statfs":{"total":386547056640,"available":383711121408,"internally_reserved":0,"allocated":1421312,"data_stored":580297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001592"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.923123+0000","last_change":"2024-09-11T04:56:20.598581+0000","last_active":"2024-09-11T05:07:58.923123+0000","last_peered":"2024-09-11T05:07:58.923123+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:07:58.923123+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12276,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217538+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:09.217538+0000","last_peered":"2024-09-11T05:08:09.217538+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:09.217538+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13938,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:59.779704+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:07:59.779704+0000","last_peered":"2024-09-11T05:07:59.779704+0000","last_clean":"2024-09-11T05:07:59.779704+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:07:59.779704+0000","last_undegraded":"2024-09-11T05:07:59.779704+0000","last_fullsized":"2024-09-11T05:07:59.779704+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17122,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.324998+0000","last_change":"2024-09-11T05:08:10.324876+0000","last_active":"2024-09-11T05:08:10.324998+0000","last_peered":"2024-09-11T05:08:10.324998+0000","last_clean":"2024-09-11T05:08:10.324998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:10.324998+0000","last_undegraded":"2024-09-11T05:08:10.324998+0000","last_fullsized":"2024-09-11T05:08:10.324998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2254,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12832,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922685+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:07:58.922685+0000","last_peered":"2024-09-11T05:07:58.922685+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:07:58.922685+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17690,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.918870+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:07:58.918870+0000","last_peered":"2024-09-11T05:07:58.918870+0000","last_clean":"2024-09-11T05:07:58.918870+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:07:58.918870+0000","last_undegraded":"2024-09-11T05:07:58.918870+0000","last_fullsized":"2024-09-11T05:07:58.918870+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12650,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217446+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:09.217446+0000","last_peered":"2024-09-11T05:08:09.217446+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:09.217446+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.919501+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:07:58.919501+0000","last_peered":"2024-09-11T05:07:58.919501+0000","last_clean":"2024-09-11T05:07:58.919501+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:07:58.919501+0000","last_undegraded":"2024-09-11T05:07:58.919501+0000","last_fullsized":"2024-09-11T05:07:58.919501+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17508,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:07:58.920018+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:07:58.920018+0000","last_peered":"2024-09-11T05:07:58.920018+0000","last_clean":"2024-09-11T05:07:58.920018+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:07:58.920018+0000","last_undegraded":"2024-09-11T05:07:58.920018+0000","last_fullsized":"2024-09-11T05:07:58.920018+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12537,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:09.217076+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:09.217076+0000","last_peered":"2024-09-11T05:08:09.217076+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:09.217076+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12904,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:07:58.922687+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:07:58.922687+0000","last_peered":"2024-09-11T05:07:58.922687+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:07:58.922687+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":148192,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161614,"num_write_kb":604172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650340,"statfs":{"total":96636764160,"available":95897948160,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":201710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721944,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649896,"statfs":{"total":96636764160,"available":95897493504,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:16.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:16.430 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:16.443 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-11T05:08:15.934395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765340,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2763825,"kb_avail":374722020,"statfs":{"total":386547056640,"available":383715348480,"internally_reserved":0,"allocated":1462272,"data_stored":592892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2830157366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12527,"num_write_kb":-50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-801,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001462"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":14085,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:12.403058+0000","last_change":"2024-09-11T05:08:08.928767+0000","last_active":"2024-09-11T05:08:12.403058+0000","last_peered":"2024-09-11T05:08:12.403058+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:12.403058+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12278,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230510+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:10.230510+0000","last_peered":"2024-09-11T05:08:10.230510+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:10.230510+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13942,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.229939+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:10.229939+0000","last_peered":"2024-09-11T05:08:10.229939+0000","last_clean":"2024-09-11T05:08:10.229939+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:10.229939+0000","last_undegraded":"2024-09-11T05:08:10.229939+0000","last_fullsized":"2024-09-11T05:08:10.229939+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17130,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:11.278519+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:11.278519+0000","last_peered":"2024-09-11T05:08:11.278519+0000","last_clean":"2024-09-11T05:08:11.278519+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:11.278519+0000","last_undegraded":"2024-09-11T05:08:11.278519+0000","last_fullsized":"2024-09-11T05:08:11.278519+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12836,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225772+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:10.225772+0000","last_peered":"2024-09-11T05:08:10.225772+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:10.225772+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17694,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:10.230619+0000","last_peered":"2024-09-11T05:08:10.230619+0000","last_clean":"2024-09-11T05:08:10.230619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:10.230619+0000","last_undegraded":"2024-09-11T05:08:10.230619+0000","last_fullsized":"2024-09-11T05:08:10.230619+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12652,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230998+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:10.230998+0000","last_peered":"2024-09-11T05:08:10.230998+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:10.230998+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230933+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:10.230933+0000","last_peered":"2024-09-11T05:08:10.230933+0000","last_clean":"2024-09-11T05:08:10.230933+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:10.230933+0000","last_undegraded":"2024-09-11T05:08:10.230933+0000","last_fullsized":"2024-09-11T05:08:10.230933+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17512,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230048+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:10.230048+0000","last_peered":"2024-09-11T05:08:10.230048+0000","last_clean":"2024-09-11T05:08:10.230048+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:10.230048+0000","last_undegraded":"2024-09-11T05:08:10.230048+0000","last_fullsized":"2024-09-11T05:08:10.230048+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12539,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230386+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:10.230386+0000","last_peered":"2024-09-11T05:08:10.230386+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:10.230386+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12908,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:10.225977+0000","last_peered":"2024-09-11T05:08:10.225977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:10.225977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577640,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794200,"statfs":{"total":96636764160,"available":96045260800,"internally_reserved":0,"allocated":417792,"data_stored":200447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591064786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721956,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649884,"statfs":{"total":96636764160,"available":95897481216,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:16.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:16.779 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-11T05:08:15.934395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765340,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2763825,"kb_avail":374722020,"statfs":{"total":386547056640,"available":383715348480,"internally_reserved":0,"allocated":1462272,"data_stored":592892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2830157366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12527,"num_write_kb":-50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-801,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001462"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":14085,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:12.403058+0000","last_change":"2024-09-11T05:08:08.928767+0000","last_active":"2024-09-11T05:08:12.403058+0000","last_peered":"2024-09-11T05:08:12.403058+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:12.403058+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12278,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230510+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:10.230510+0000","last_peered":"2024-09-11T05:08:10.230510+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:10.230510+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13942,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.229939+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:10.229939+0000","last_peered":"2024-09-11T05:08:10.229939+0000","last_clean":"2024-09-11T05:08:10.229939+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:10.229939+0000","last_undegraded":"2024-09-11T05:08:10.229939+0000","last_fullsized":"2024-09-11T05:08:10.229939+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17130,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:11.278519+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:11.278519+0000","last_peered":"2024-09-11T05:08:11.278519+0000","last_clean":"2024-09-11T05:08:11.278519+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:11.278519+0000","last_undegraded":"2024-09-11T05:08:11.278519+0000","last_fullsized":"2024-09-11T05:08:11.278519+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12836,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225772+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:10.225772+0000","last_peered":"2024-09-11T05:08:10.225772+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:10.225772+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17694,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:10.230619+0000","last_peered":"2024-09-11T05:08:10.230619+0000","last_clean":"2024-09-11T05:08:10.230619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:10.230619+0000","last_undegraded":"2024-09-11T05:08:10.230619+0000","last_fullsized":"2024-09-11T05:08:10.230619+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12652,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230998+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:10.230998+0000","last_peered":"2024-09-11T05:08:10.230998+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:10.230998+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230933+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:10.230933+0000","last_peered":"2024-09-11T05:08:10.230933+0000","last_clean":"2024-09-11T05:08:10.230933+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:10.230933+0000","last_undegraded":"2024-09-11T05:08:10.230933+0000","last_fullsized":"2024-09-11T05:08:10.230933+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17512,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230048+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:10.230048+0000","last_peered":"2024-09-11T05:08:10.230048+0000","last_clean":"2024-09-11T05:08:10.230048+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:10.230048+0000","last_undegraded":"2024-09-11T05:08:10.230048+0000","last_fullsized":"2024-09-11T05:08:10.230048+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12539,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230386+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:10.230386+0000","last_peered":"2024-09-11T05:08:10.230386+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:10.230386+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12908,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:10.225977+0000","last_peered":"2024-09-11T05:08:10.225977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:10.225977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577640,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794200,"statfs":{"total":96636764160,"available":96045260800,"internally_reserved":0,"allocated":417792,"data_stored":200447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591064786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721956,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649884,"statfs":{"total":96636764160,"available":95897481216,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:16.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:16.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12622, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.224830+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:10.224830+0000', 'last_peered': '2024-09-11T05:08:10.224830+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:10.224830+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:16.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:08:16.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12736", 'reported_seq': 14085, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:12.403058+0000', 'last_change': '2024-09-11T05:08:08.928767+0000', 'last_active': '2024-09-11T05:08:12.403058+0000', 'last_peered': '2024-09-11T05:08:12.403058+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:08:12.403058+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1650, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 633, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1648}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:16.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:16.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12278, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230510+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:10.230510+0000', 'last_peered': '2024-09-11T05:08:10.230510+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:10.230510+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:16.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:16.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12759, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.222630+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:10.222630+0000', 'last_peered': '2024-09-11T05:08:10.222630+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:10.222630+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12836, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.225772+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:08:10.225772+0000', 'last_peered': '2024-09-11T05:08:10.225772+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:08:10.225772+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12921, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.223741+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:10.223741+0000', 'last_peered': '2024-09-11T05:08:10.223741+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:10.223741+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 12652, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230998+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:08:10.230998+0000', 'last_peered': '2024-09-11T05:08:10.230998+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:08:10.230998+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2304}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12781, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.225785+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:10.225785+0000', 'last_peered': '2024-09-11T05:08:10.225785+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:10.225785+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12710, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.222777+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:08:10.222777+0000', 'last_peered': '2024-09-11T05:08:10.222777+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:10.222777+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:16.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:16.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12539, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230386+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:10.230386+0000', 'last_peered': '2024-09-11T05:08:10.230386+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:10.230386+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:16.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:16.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12908, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.225977+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:10.225977+0000', 'last_peered': '2024-09-11T05:08:10.225977+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:10.225977+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:16.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:17.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:17.216 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24595/80 objects degraded (30743.750%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":824,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":134720,"bytes_used":2831708160,"bytes_avail":383715348480,"bytes_total":386547056640,"degraded_objects":24595,"degraded_total":80,"degraded_ratio":307.4375},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:17.229 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:08:17.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:17.540 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:17.541 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:17.554 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-11T05:08:15.934395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765340,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2763825,"kb_avail":374722020,"statfs":{"total":386547056640,"available":383715348480,"internally_reserved":0,"allocated":1462272,"data_stored":592892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2830157366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12527,"num_write_kb":-50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-801,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001462"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":14085,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:12.403058+0000","last_change":"2024-09-11T05:08:08.928767+0000","last_active":"2024-09-11T05:08:12.403058+0000","last_peered":"2024-09-11T05:08:12.403058+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:12.403058+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12278,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230510+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:10.230510+0000","last_peered":"2024-09-11T05:08:10.230510+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:10.230510+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13942,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.229939+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:10.229939+0000","last_peered":"2024-09-11T05:08:10.229939+0000","last_clean":"2024-09-11T05:08:10.229939+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:10.229939+0000","last_undegraded":"2024-09-11T05:08:10.229939+0000","last_fullsized":"2024-09-11T05:08:10.229939+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17130,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:11.278519+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:11.278519+0000","last_peered":"2024-09-11T05:08:11.278519+0000","last_clean":"2024-09-11T05:08:11.278519+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:11.278519+0000","last_undegraded":"2024-09-11T05:08:11.278519+0000","last_fullsized":"2024-09-11T05:08:11.278519+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12836,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225772+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:10.225772+0000","last_peered":"2024-09-11T05:08:10.225772+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:10.225772+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17694,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:10.230619+0000","last_peered":"2024-09-11T05:08:10.230619+0000","last_clean":"2024-09-11T05:08:10.230619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:10.230619+0000","last_undegraded":"2024-09-11T05:08:10.230619+0000","last_fullsized":"2024-09-11T05:08:10.230619+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12652,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230998+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:10.230998+0000","last_peered":"2024-09-11T05:08:10.230998+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:10.230998+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230933+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:10.230933+0000","last_peered":"2024-09-11T05:08:10.230933+0000","last_clean":"2024-09-11T05:08:10.230933+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:10.230933+0000","last_undegraded":"2024-09-11T05:08:10.230933+0000","last_fullsized":"2024-09-11T05:08:10.230933+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17512,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230048+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:10.230048+0000","last_peered":"2024-09-11T05:08:10.230048+0000","last_clean":"2024-09-11T05:08:10.230048+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:10.230048+0000","last_undegraded":"2024-09-11T05:08:10.230048+0000","last_fullsized":"2024-09-11T05:08:10.230048+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12539,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230386+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:10.230386+0000","last_peered":"2024-09-11T05:08:10.230386+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:10.230386+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12908,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:10.225977+0000","last_peered":"2024-09-11T05:08:10.225977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:10.225977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577640,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794200,"statfs":{"total":96636764160,"available":96045260800,"internally_reserved":0,"allocated":417792,"data_stored":200447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":591064786},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721956,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649884,"statfs":{"total":96636764160,"available":95897481216,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:20.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:20.867 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:20.867 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:20.879 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-11T05:08:19.935073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769500,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717860,"statfs":{"total":386547056640,"available":383711088640,"internally_reserved":0,"allocated":1462272,"data_stored":592892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12527,"num_write_kb":-50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001763"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":15771,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:17.407037+0000","last_change":"2024-09-11T05:08:08.928767+0000","last_active":"2024-09-11T05:08:17.407037+0000","last_peered":"2024-09-11T05:08:17.407037+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:17.407037+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12278,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230510+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:10.230510+0000","last_peered":"2024-09-11T05:08:10.230510+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:10.230510+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13942,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.229939+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:10.229939+0000","last_peered":"2024-09-11T05:08:10.229939+0000","last_clean":"2024-09-11T05:08:10.229939+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:10.229939+0000","last_undegraded":"2024-09-11T05:08:10.229939+0000","last_fullsized":"2024-09-11T05:08:10.229939+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17130,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:11.278519+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:11.278519+0000","last_peered":"2024-09-11T05:08:11.278519+0000","last_clean":"2024-09-11T05:08:11.278519+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:11.278519+0000","last_undegraded":"2024-09-11T05:08:11.278519+0000","last_fullsized":"2024-09-11T05:08:11.278519+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12836,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225772+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:10.225772+0000","last_peered":"2024-09-11T05:08:10.225772+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:10.225772+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17694,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:10.230619+0000","last_peered":"2024-09-11T05:08:10.230619+0000","last_clean":"2024-09-11T05:08:10.230619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:10.230619+0000","last_undegraded":"2024-09-11T05:08:10.230619+0000","last_fullsized":"2024-09-11T05:08:10.230619+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12652,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230998+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:10.230998+0000","last_peered":"2024-09-11T05:08:10.230998+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:10.230998+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230983+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:10.230983+0000","last_peered":"2024-09-11T05:08:10.230983+0000","last_clean":"2024-09-11T05:08:10.230983+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:10.230983+0000","last_undegraded":"2024-09-11T05:08:10.230983+0000","last_fullsized":"2024-09-11T05:08:10.230983+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17512,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230048+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:10.230048+0000","last_peered":"2024-09-11T05:08:10.230048+0000","last_clean":"2024-09-11T05:08:10.230048+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:10.230048+0000","last_undegraded":"2024-09-11T05:08:10.230048+0000","last_fullsized":"2024-09-11T05:08:10.230048+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12539,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230386+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:10.230386+0000","last_peered":"2024-09-11T05:08:10.230386+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:10.230386+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12908,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:10.225977+0000","last_peered":"2024-09-11T05:08:10.225977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:10.225977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":417792,"data_stored":200447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721956,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649884,"statfs":{"total":96636764160,"available":95897481216,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:20.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:21.232 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:21.233 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:21.245 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-11T05:08:19.935073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769500,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717860,"statfs":{"total":386547056640,"available":383711088640,"internally_reserved":0,"allocated":1462272,"data_stored":592892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12527,"num_write_kb":-50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001763"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":15771,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:17.407037+0000","last_change":"2024-09-11T05:08:08.928767+0000","last_active":"2024-09-11T05:08:17.407037+0000","last_peered":"2024-09-11T05:08:17.407037+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:17.407037+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12278,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230510+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:10.230510+0000","last_peered":"2024-09-11T05:08:10.230510+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:10.230510+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13942,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.229939+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:10.229939+0000","last_peered":"2024-09-11T05:08:10.229939+0000","last_clean":"2024-09-11T05:08:10.229939+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:10.229939+0000","last_undegraded":"2024-09-11T05:08:10.229939+0000","last_fullsized":"2024-09-11T05:08:10.229939+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17130,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:11.278519+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:11.278519+0000","last_peered":"2024-09-11T05:08:11.278519+0000","last_clean":"2024-09-11T05:08:11.278519+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:11.278519+0000","last_undegraded":"2024-09-11T05:08:11.278519+0000","last_fullsized":"2024-09-11T05:08:11.278519+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12836,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225772+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:10.225772+0000","last_peered":"2024-09-11T05:08:10.225772+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:10.225772+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17694,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:10.230619+0000","last_peered":"2024-09-11T05:08:10.230619+0000","last_clean":"2024-09-11T05:08:10.230619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:10.230619+0000","last_undegraded":"2024-09-11T05:08:10.230619+0000","last_fullsized":"2024-09-11T05:08:10.230619+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12652,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230998+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:10.230998+0000","last_peered":"2024-09-11T05:08:10.230998+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:10.230998+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230983+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:10.230983+0000","last_peered":"2024-09-11T05:08:10.230983+0000","last_clean":"2024-09-11T05:08:10.230983+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:10.230983+0000","last_undegraded":"2024-09-11T05:08:10.230983+0000","last_fullsized":"2024-09-11T05:08:10.230983+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17512,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230048+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:10.230048+0000","last_peered":"2024-09-11T05:08:10.230048+0000","last_clean":"2024-09-11T05:08:10.230048+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:10.230048+0000","last_undegraded":"2024-09-11T05:08:10.230048+0000","last_fullsized":"2024-09-11T05:08:10.230048+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12539,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230386+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:10.230386+0000","last_peered":"2024-09-11T05:08:10.230386+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:10.230386+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12908,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:10.225977+0000","last_peered":"2024-09-11T05:08:10.225977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:10.225977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":417792,"data_stored":200447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721956,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649884,"statfs":{"total":96636764160,"available":95897481216,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:21.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12622, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.224830+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:10.224830+0000', 'last_peered': '2024-09-11T05:08:10.224830+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:10.224830+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12736", 'reported_seq': 15771, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:17.407037+0000', 'last_change': '2024-09-11T05:08:08.928767+0000', 'last_active': '2024-09-11T05:08:17.407037+0000', 'last_peered': '2024-09-11T05:08:17.407037+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:08:17.407037+0000', 'last_undegraded': '2024-09-11T04:56:20.262919+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 807, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1476, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12278, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230510+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:10.230510+0000', 'last_peered': '2024-09-11T05:08:10.230510+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:10.230510+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12759, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.222630+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:10.222630+0000', 'last_peered': '2024-09-11T05:08:10.222630+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:10.222630+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:21.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12836, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.225772+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:08:10.225772+0000', 'last_peered': '2024-09-11T05:08:10.225772+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:08:10.225772+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12921, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.223741+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:10.223741+0000', 'last_peered': '2024-09-11T05:08:10.223741+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:10.223741+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 12652, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230998+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:08:10.230998+0000', 'last_peered': '2024-09-11T05:08:10.230998+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:08:10.230998+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2304}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12781, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.225785+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:10.225785+0000', 'last_peered': '2024-09-11T05:08:10.225785+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:10.225785+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12710, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.222777+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:08:10.222777+0000', 'last_peered': '2024-09-11T05:08:10.222777+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:10.222777+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12539, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230386+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:10.230386+0000', 'last_peered': '2024-09-11T05:08:10.230386+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:10.230386+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:21.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:21.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12908, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.225977+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:10.225977+0000', 'last_peered': '2024-09-11T05:08:10.225977+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:10.225977+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:21.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:21.720 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:21.720 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23752/80 objects degraded (29690.000%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":828,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":134720,"bytes_used":2835968000,"bytes_avail":383711088640,"bytes_total":386547056640,"degraded_objects":23752,"degraded_total":80,"degraded_ratio":296.89999999999998,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:21.735 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:08:21.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:22.081 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:22.082 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:22.095 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-11T05:08:21.935448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769500,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2767985,"kb_avail":374717860,"statfs":{"total":386547056640,"available":383711088640,"internally_reserved":0,"allocated":1462272,"data_stored":592892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2834417206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12527,"num_write_kb":-50117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-801,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001749"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":15771,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:17.407037+0000","last_change":"2024-09-11T05:08:08.928767+0000","last_active":"2024-09-11T05:08:17.407037+0000","last_peered":"2024-09-11T05:08:17.407037+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:17.407037+0000","last_undegraded":"2024-09-11T04:56:20.262919+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12278,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230510+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:10.230510+0000","last_peered":"2024-09-11T05:08:10.230510+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:10.230510+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13942,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.229939+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:10.229939+0000","last_peered":"2024-09-11T05:08:10.229939+0000","last_clean":"2024-09-11T05:08:10.229939+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:10.229939+0000","last_undegraded":"2024-09-11T05:08:10.229939+0000","last_fullsized":"2024-09-11T05:08:10.229939+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17130,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:11.278519+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:11.278519+0000","last_peered":"2024-09-11T05:08:11.278519+0000","last_clean":"2024-09-11T05:08:11.278519+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:11.278519+0000","last_undegraded":"2024-09-11T05:08:11.278519+0000","last_fullsized":"2024-09-11T05:08:11.278519+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12836,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225772+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:10.225772+0000","last_peered":"2024-09-11T05:08:10.225772+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:10.225772+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17694,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230619+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:10.230619+0000","last_peered":"2024-09-11T05:08:10.230619+0000","last_clean":"2024-09-11T05:08:10.230619+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:10.230619+0000","last_undegraded":"2024-09-11T05:08:10.230619+0000","last_fullsized":"2024-09-11T05:08:10.230619+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12652,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230998+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:10.230998+0000","last_peered":"2024-09-11T05:08:10.230998+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:10.230998+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":16999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230983+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:10.230983+0000","last_peered":"2024-09-11T05:08:10.230983+0000","last_clean":"2024-09-11T05:08:10.230983+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:10.230983+0000","last_undegraded":"2024-09-11T05:08:10.230983+0000","last_fullsized":"2024-09-11T05:08:10.230983+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17512,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:10.230048+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:10.230048+0000","last_peered":"2024-09-11T05:08:10.230048+0000","last_clean":"2024-09-11T05:08:10.230048+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:10.230048+0000","last_undegraded":"2024-09-11T05:08:10.230048+0000","last_fullsized":"2024-09-11T05:08:10.230048+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12539,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230386+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:10.230386+0000","last_peered":"2024-09-11T05:08:10.230386+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:10.230386+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12908,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225977+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:10.225977+0000","last_peered":"2024-09-11T05:08:10.225977+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:10.225977+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":417792,"data_stored":200447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":595324626},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721956,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649884,"statfs":{"total":96636764160,"available":95897481216,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:25.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:25.419 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:25.432 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-11T05:08:23.935933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779244,"kb_used_data":1436,"kb_used_omap":14,"kb_used_meta":2777713,"kb_avail":374708116,"statfs":{"total":386547056640,"available":383701110784,"internally_reserved":0,"allocated":1470464,"data_stored":591576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2844378678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1650,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001538"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":42,"state":"peering","last_fresh":"2024-09-11T05:08:20.775881+0000","last_change":"2024-09-11T05:08:22.085906+0000","last_active":"2024-09-11T05:08:22.085906+0000","last_peered":"2024-09-11T05:08:20.775881+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:20.775881+0000","last_undegraded":"2024-09-11T05:08:20.775881+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12278,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230510+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:10.230510+0000","last_peered":"2024-09-11T05:08:10.230510+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:10.230510+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13944,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.259700+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:21.259700+0000","last_peered":"2024-09-11T05:08:21.259700+0000","last_clean":"2024-09-11T05:08:21.259700+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:21.259700+0000","last_undegraded":"2024-09-11T05:08:21.259700+0000","last_fullsized":"2024-09-11T05:08:21.259700+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17130,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:11.278519+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:11.278519+0000","last_peered":"2024-09-11T05:08:11.278519+0000","last_clean":"2024-09-11T05:08:11.278519+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:11.278519+0000","last_undegraded":"2024-09-11T05:08:11.278519+0000","last_fullsized":"2024-09-11T05:08:11.278519+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17696,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.260182+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:21.260182+0000","last_peered":"2024-09-11T05:08:21.260182+0000","last_clean":"2024-09-11T05:08:21.260182+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:21.260182+0000","last_undegraded":"2024-09-11T05:08:21.260182+0000","last_fullsized":"2024-09-11T05:08:21.260182+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12652,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230998+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:10.230998+0000","last_peered":"2024-09-11T05:08:10.230998+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:10.230998+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17001,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.260379+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:21.260379+0000","last_peered":"2024-09-11T05:08:21.260379+0000","last_clean":"2024-09-11T05:08:21.260379+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:21.260379+0000","last_undegraded":"2024-09-11T05:08:21.260379+0000","last_fullsized":"2024-09-11T05:08:21.260379+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17514,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.259841+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:21.259841+0000","last_peered":"2024-09-11T05:08:21.259841+0000","last_clean":"2024-09-11T05:08:21.259841+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:21.259841+0000","last_undegraded":"2024-09-11T05:08:21.259841+0000","last_fullsized":"2024-09-11T05:08:21.259841+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12539,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230386+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:10.230386+0000","last_peered":"2024-09-11T05:08:10.230386+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:10.230386+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591532,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780308,"statfs":{"total":96636764160,"available":96031035392,"internally_reserved":0,"allocated":417792,"data_stored":195693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605286098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721968,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649872,"statfs":{"total":96636764160,"available":95897468928,"internally_reserved":0,"allocated":356352,"data_stored":134253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:25.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:25.758 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:25.759 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:25.771 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-11T05:08:23.935933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779244,"kb_used_data":1436,"kb_used_omap":14,"kb_used_meta":2777713,"kb_avail":374708116,"statfs":{"total":386547056640,"available":383701110784,"internally_reserved":0,"allocated":1470464,"data_stored":591576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2844378678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1650,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001538"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.224830+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:10.224830+0000","last_peered":"2024-09-11T05:08:10.224830+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:10.224830+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":42,"state":"peering","last_fresh":"2024-09-11T05:08:20.775881+0000","last_change":"2024-09-11T05:08:22.085906+0000","last_active":"2024-09-11T05:08:22.085906+0000","last_peered":"2024-09-11T05:08:20.775881+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:20.775881+0000","last_undegraded":"2024-09-11T05:08:20.775881+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12278,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230510+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:10.230510+0000","last_peered":"2024-09-11T05:08:10.230510+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:10.230510+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13944,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.259700+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:21.259700+0000","last_peered":"2024-09-11T05:08:21.259700+0000","last_clean":"2024-09-11T05:08:21.259700+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:21.259700+0000","last_undegraded":"2024-09-11T05:08:21.259700+0000","last_fullsized":"2024-09-11T05:08:21.259700+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17130,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-11T05:08:11.278519+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:11.278519+0000","last_peered":"2024-09-11T05:08:11.278519+0000","last_clean":"2024-09-11T05:08:11.278519+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:11.278519+0000","last_undegraded":"2024-09-11T05:08:11.278519+0000","last_fullsized":"2024-09-11T05:08:11.278519+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12759,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222630+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:10.222630+0000","last_peered":"2024-09-11T05:08:10.222630+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:10.222630+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17696,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.260182+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:21.260182+0000","last_peered":"2024-09-11T05:08:21.260182+0000","last_clean":"2024-09-11T05:08:21.260182+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:21.260182+0000","last_undegraded":"2024-09-11T05:08:21.260182+0000","last_fullsized":"2024-09-11T05:08:21.260182+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12921,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.223741+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:10.223741+0000","last_peered":"2024-09-11T05:08:10.223741+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:10.223741+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12652,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230998+0000","last_change":"2024-09-11T04:56:20.592440+0000","last_active":"2024-09-11T05:08:10.230998+0000","last_peered":"2024-09-11T05:08:10.230998+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:10.230998+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.225785+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:10.225785+0000","last_peered":"2024-09-11T05:08:10.225785+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:10.225785+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17001,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.260379+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:21.260379+0000","last_peered":"2024-09-11T05:08:21.260379+0000","last_clean":"2024-09-11T05:08:21.260379+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:21.260379+0000","last_undegraded":"2024-09-11T05:08:21.260379+0000","last_fullsized":"2024-09-11T05:08:21.260379+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17514,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.259841+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:21.259841+0000","last_peered":"2024-09-11T05:08:21.259841+0000","last_clean":"2024-09-11T05:08:21.259841+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:21.259841+0000","last_undegraded":"2024-09-11T05:08:21.259841+0000","last_fullsized":"2024-09-11T05:08:21.259841+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12710,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.222777+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:10.222777+0000","last_peered":"2024-09-11T05:08:10.222777+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:10.222777+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12539,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:10.230386+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:10.230386+0000","last_peered":"2024-09-11T05:08:10.230386+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:10.230386+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744232,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627608,"statfs":{"total":96636764160,"available":95874670592,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591532,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780308,"statfs":{"total":96636764160,"available":96031035392,"internally_reserved":0,"allocated":417792,"data_stored":195693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605286098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721968,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649872,"statfs":{"total":96636764160,"available":95897468928,"internally_reserved":0,"allocated":356352,"data_stored":134253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:25.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:25.772 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12622, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.224830+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:10.224830+0000', 'last_peered': '2024-09-11T05:08:10.224830+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:10.224830+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:25.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-11T05:08:25.772 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12736", 'reported_seq': 12860, 'reported_epoch': 42, 'state': 'peering', 'last_fresh': '2024-09-11T05:08:20.775881+0000', 'last_change': '2024-09-11T05:08:22.085906+0000', 'last_active': '2024-09-11T05:08:22.085906+0000', 'last_peered': '2024-09-11T05:08:20.775881+0000', 'last_clean': '2024-09-11T04:55:08.191690+0000', 'last_became_active': '2024-09-11T04:56:20.591403+0000', 'last_became_peered': '2024-09-11T04:56:20.591403+0000', 'last_unstale': '2024-09-11T05:08:20.775881+0000', 'last_undegraded': '2024-09-11T05:08:20.775881+0000', 'last_fullsized': '2024-09-11T04:56:20.223582+0000', 'mapping_epoch': 42, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12734, 'num_write_kb': 50948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2283, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [1, 0, 3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-11T05:08:25.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:25.772 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12278, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230510+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:10.230510+0000', 'last_peered': '2024-09-11T05:08:10.230510+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:10.230510+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:25.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:25.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12759, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.222630+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:10.222630+0000', 'last_peered': '2024-09-11T05:08:10.222630+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:10.222630+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:25.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12840, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.258452+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:08:22.258452+0000', 'last_peered': '2024-09-11T05:08:22.258452+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:08:22.258452+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12921, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.223741+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:10.223741+0000', 'last_peered': '2024-09-11T05:08:10.223741+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:10.223741+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 12652, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230998+0000', 'last_change': '2024-09-11T04:56:20.592440+0000', 'last_active': '2024-09-11T05:08:10.230998+0000', 'last_peered': '2024-09-11T05:08:10.230998+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:08:10.230998+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2304}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12781, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.225785+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:10.225785+0000', 'last_peered': '2024-09-11T05:08:10.225785+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:10.225785+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12710, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.222777+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:08:10.222777+0000', 'last_peered': '2024-09-11T05:08:10.222777+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:10.222777+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:25.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:25.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12539, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:10.230386+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:10.230386+0000', 'last_peered': '2024-09-11T05:08:10.230386+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:10.230386+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:25.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:25.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12912, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.259413+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:22.259413+0000', 'last_peered': '2024-09-11T05:08:22.259413+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:22.259413+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:25.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:26.158 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:26.158 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22945/80 objects degraded (28681.250%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":833,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":134720,"bytes_used":2845945856,"bytes_avail":383701110784,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":22945,"degraded_total":80,"degraded_ratio":286.8125,"recovering_objects_per_sec":206,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1650,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:26.172 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:08:26.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:26.487 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:26.488 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:26.500 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-11T05:08:25.936270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779276,"kb_used_data":1468,"kb_used_omap":14,"kb_used_meta":2777713,"kb_avail":374708084,"statfs":{"total":386547056640,"available":383701078016,"internally_reserved":0,"allocated":1503232,"data_stored":605286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2844378678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001564"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12626,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263963+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:22.263963+0000","last_peered":"2024-09-11T05:08:22.263963+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:22.263963+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12860,"reported_epoch":42,"state":"peering","last_fresh":"2024-09-11T05:08:20.775881+0000","last_change":"2024-09-11T05:08:22.085906+0000","last_active":"2024-09-11T05:08:22.085906+0000","last_peered":"2024-09-11T05:08:20.775881+0000","last_clean":"2024-09-11T04:55:08.191690+0000","last_became_active":"2024-09-11T04:56:20.591403+0000","last_became_peered":"2024-09-11T04:56:20.591403+0000","last_unstale":"2024-09-11T05:08:20.775881+0000","last_undegraded":"2024-09-11T05:08:20.775881+0000","last_fullsized":"2024-09-11T04:56:20.223582+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12734,"num_write_kb":50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12282,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.261977+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:22.261977+0000","last_peered":"2024-09-11T05:08:22.261977+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:22.261977+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13944,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.259700+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:21.259700+0000","last_peered":"2024-09-11T05:08:21.259700+0000","last_clean":"2024-09-11T05:08:21.259700+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:21.259700+0000","last_undegraded":"2024-09-11T05:08:21.259700+0000","last_fullsized":"2024-09-11T05:08:21.259700+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17134,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.262506+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:22.262506+0000","last_peered":"2024-09-11T05:08:22.262506+0000","last_clean":"2024-09-11T05:08:22.262506+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:22.262506+0000","last_undegraded":"2024-09-11T05:08:22.262506+0000","last_fullsized":"2024-09-11T05:08:22.262506+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12763,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.264715+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:22.264715+0000","last_peered":"2024-09-11T05:08:22.264715+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:22.264715+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17696,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.260182+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:21.260182+0000","last_peered":"2024-09-11T05:08:21.260182+0000","last_clean":"2024-09-11T05:08:21.260182+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:21.260182+0000","last_undegraded":"2024-09-11T05:08:21.260182+0000","last_fullsized":"2024-09-11T05:08:21.260182+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262595+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:22.262595+0000","last_peered":"2024-09-11T05:08:22.262595+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:22.262595+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":14137,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:25.069541+0000","last_change":"2024-09-11T05:08:20.776610+0000","last_active":"2024-09-11T05:08:25.069541+0000","last_peered":"2024-09-11T05:08:25.069541+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:25.069541+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12785,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:22.263962+0000","last_peered":"2024-09-11T05:08:22.263962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:22.263962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17001,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.260379+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:21.260379+0000","last_peered":"2024-09-11T05:08:21.260379+0000","last_clean":"2024-09-11T05:08:21.260379+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:21.260379+0000","last_undegraded":"2024-09-11T05:08:21.260379+0000","last_fullsized":"2024-09-11T05:08:21.260379+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17514,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-11T05:08:21.259841+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:21.259841+0000","last_peered":"2024-09-11T05:08:21.259841+0000","last_clean":"2024-09-11T05:08:21.259841+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:21.259841+0000","last_undegraded":"2024-09-11T05:08:21.259841+0000","last_fullsized":"2024-09-11T05:08:21.259841+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:22.262644+0000","last_peered":"2024-09-11T05:08:22.262644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:22.262644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12544,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263265+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:22.263265+0000","last_peered":"2024-09-11T05:08:22.263265+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:22.263265+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":134720,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149087,"num_write_kb":554055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744248,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627592,"statfs":{"total":96636764160,"available":95874654208,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721528,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650312,"statfs":{"total":96636764160,"available":95897919488,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591532,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780308,"statfs":{"total":96636764160,"available":96031035392,"internally_reserved":0,"allocated":417792,"data_stored":195693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":605286098},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721968,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649872,"statfs":{"total":96636764160,"available":95897468928,"internally_reserved":0,"allocated":356352,"data_stored":134253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:29.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:29.834 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:29.848 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-11T05:08:27.936713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805652,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681708,"statfs":{"total":386547056640,"available":383674068992,"internally_reserved":0,"allocated":1519616,"data_stored":612120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12734,"num_write_kb":-50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001639"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12626,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263963+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:22.263963+0000","last_peered":"2024-09-11T05:08:22.263963+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:22.263963+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12879,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.475708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:22.475708+0000","last_peered":"2024-09-11T05:08:22.475708+0000","last_clean":"2024-09-11T05:08:22.475708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:22.475708+0000","last_undegraded":"2024-09-11T05:08:22.475708+0000","last_fullsized":"2024-09-11T05:08:22.475708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12282,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.261977+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:22.261977+0000","last_peered":"2024-09-11T05:08:22.261977+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:22.261977+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13946,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257737+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:22.257737+0000","last_peered":"2024-09-11T05:08:22.257737+0000","last_clean":"2024-09-11T05:08:22.257737+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:22.257737+0000","last_undegraded":"2024-09-11T05:08:22.257737+0000","last_fullsized":"2024-09-11T05:08:22.257737+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17134,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.262506+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:22.262506+0000","last_peered":"2024-09-11T05:08:22.262506+0000","last_clean":"2024-09-11T05:08:22.262506+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:22.262506+0000","last_undegraded":"2024-09-11T05:08:22.262506+0000","last_fullsized":"2024-09-11T05:08:22.262506+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12763,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.264715+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:22.264715+0000","last_peered":"2024-09-11T05:08:22.264715+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:22.264715+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17698,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258084+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:22.258084+0000","last_peered":"2024-09-11T05:08:22.258084+0000","last_clean":"2024-09-11T05:08:22.258084+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:22.258084+0000","last_undegraded":"2024-09-11T05:08:22.258084+0000","last_fullsized":"2024-09-11T05:08:22.258084+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262595+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:22.262595+0000","last_peered":"2024-09-11T05:08:22.262595+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:22.262595+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":14137,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:25.069541+0000","last_change":"2024-09-11T05:08:20.776610+0000","last_active":"2024-09-11T05:08:25.069541+0000","last_peered":"2024-09-11T05:08:25.069541+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:25.069541+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12785,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:22.263962+0000","last_peered":"2024-09-11T05:08:22.263962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:22.263962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258195+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:22.258195+0000","last_peered":"2024-09-11T05:08:22.258195+0000","last_clean":"2024-09-11T05:08:22.258195+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:22.258195+0000","last_undegraded":"2024-09-11T05:08:22.258195+0000","last_fullsized":"2024-09-11T05:08:22.258195+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17516,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257742+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:22.257742+0000","last_peered":"2024-09-11T05:08:22.257742+0000","last_clean":"2024-09-11T05:08:22.257742+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:22.257742+0000","last_undegraded":"2024-09-11T05:08:22.257742+0000","last_fullsized":"2024-09-11T05:08:22.257742+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:22.262644+0000","last_peered":"2024-09-11T05:08:22.262644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:22.262644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12544,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263265+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:22.263265+0000","last_peered":"2024-09-11T05:08:22.263265+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:22.263265+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744248,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627592,"statfs":{"total":96636764160,"available":95874654208,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721528,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650312,"statfs":{"total":96636764160,"available":95897919488,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617904,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753936,"statfs":{"total":96636764160,"available":96004030464,"internally_reserved":0,"allocated":425984,"data_stored":199110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721972,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649868,"statfs":{"total":96636764160,"available":95897464832,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:29.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:30.157 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:30.170 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-11T05:08:29.937056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805652,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681708,"statfs":{"total":386547056640,"available":383674068992,"internally_reserved":0,"allocated":1519616,"data_stored":612120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12734,"num_write_kb":-50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.686253"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12626,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263963+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:22.263963+0000","last_peered":"2024-09-11T05:08:22.263963+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:22.263963+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12879,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.475708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:22.475708+0000","last_peered":"2024-09-11T05:08:22.475708+0000","last_clean":"2024-09-11T05:08:22.475708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:22.475708+0000","last_undegraded":"2024-09-11T05:08:22.475708+0000","last_fullsized":"2024-09-11T05:08:22.475708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12282,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.261977+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:22.261977+0000","last_peered":"2024-09-11T05:08:22.261977+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:22.261977+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13946,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257737+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:22.257737+0000","last_peered":"2024-09-11T05:08:22.257737+0000","last_clean":"2024-09-11T05:08:22.257737+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:22.257737+0000","last_undegraded":"2024-09-11T05:08:22.257737+0000","last_fullsized":"2024-09-11T05:08:22.257737+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17134,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.262506+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:22.262506+0000","last_peered":"2024-09-11T05:08:22.262506+0000","last_clean":"2024-09-11T05:08:22.262506+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:22.262506+0000","last_undegraded":"2024-09-11T05:08:22.262506+0000","last_fullsized":"2024-09-11T05:08:22.262506+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12763,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.264715+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:22.264715+0000","last_peered":"2024-09-11T05:08:22.264715+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:22.264715+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17698,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258084+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:22.258084+0000","last_peered":"2024-09-11T05:08:22.258084+0000","last_clean":"2024-09-11T05:08:22.258084+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:22.258084+0000","last_undegraded":"2024-09-11T05:08:22.258084+0000","last_fullsized":"2024-09-11T05:08:22.258084+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262595+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:22.262595+0000","last_peered":"2024-09-11T05:08:22.262595+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:22.262595+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":14137,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:25.069541+0000","last_change":"2024-09-11T05:08:20.776610+0000","last_active":"2024-09-11T05:08:25.069541+0000","last_peered":"2024-09-11T05:08:25.069541+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:25.069541+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12785,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:22.263962+0000","last_peered":"2024-09-11T05:08:22.263962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:22.263962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258195+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:22.258195+0000","last_peered":"2024-09-11T05:08:22.258195+0000","last_clean":"2024-09-11T05:08:22.258195+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:22.258195+0000","last_undegraded":"2024-09-11T05:08:22.258195+0000","last_fullsized":"2024-09-11T05:08:22.258195+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17516,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257742+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:22.257742+0000","last_peered":"2024-09-11T05:08:22.257742+0000","last_clean":"2024-09-11T05:08:22.257742+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:22.257742+0000","last_undegraded":"2024-09-11T05:08:22.257742+0000","last_fullsized":"2024-09-11T05:08:22.257742+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:22.262644+0000","last_peered":"2024-09-11T05:08:22.262644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:22.262644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12544,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263265+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:22.263265+0000","last_peered":"2024-09-11T05:08:22.263265+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:22.263265+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744248,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627592,"statfs":{"total":96636764160,"available":95874654208,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721528,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650312,"statfs":{"total":96636764160,"available":95897919488,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617904,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753936,"statfs":{"total":96636764160,"available":96004030464,"internally_reserved":0,"allocated":425984,"data_stored":199110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721972,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649868,"statfs":{"total":96636764160,"available":95897464832,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:30.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:30.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12626, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.263963+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:22.263963+0000', 'last_peered': '2024-09-11T05:08:22.263963+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:22.263963+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:30.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:30.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12282, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.261977+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:22.261977+0000', 'last_peered': '2024-09-11T05:08:22.261977+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:22.261977+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:30.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:30.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12763, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.264715+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:22.264715+0000', 'last_peered': '2024-09-11T05:08:22.264715+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:22.264715+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:30.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:30.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12840, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.258452+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:08:22.258452+0000', 'last_peered': '2024-09-11T05:08:22.258452+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:08:22.258452+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:30.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:30.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12925, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.262595+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:22.262595+0000', 'last_peered': '2024-09-11T05:08:22.262595+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:22.262595+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:30.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:08:30.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 14137, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:25.069541+0000', 'last_change': '2024-09-11T05:08:20.776610+0000', 'last_active': '2024-09-11T05:08:25.069541+0000', 'last_peered': '2024-09-11T05:08:25.069541+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:08:25.069541+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1568, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 754, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1566}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:30.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:30.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12785, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.263962+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:22.263962+0000', 'last_peered': '2024-09-11T05:08:22.263962+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:22.263962+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:30.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:30.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12714, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.262644+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:08:22.262644+0000', 'last_peered': '2024-09-11T05:08:22.262644+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:22.262644+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:30.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:30.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12544, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.263265+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:22.263265+0000', 'last_peered': '2024-09-11T05:08:22.263265+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:22.263265+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:30.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:30.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12912, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.259413+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:22.259413+0000', 'last_peered': '2024-09-11T05:08:22.259413+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:22.259413+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:30.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:30.575 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:30.575 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22205/72 objects degraded (30840.278%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":837,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":121248,"bytes_used":2872987648,"bytes_avail":383674068992,"bytes_total":386547056640,"degraded_objects":22205,"degraded_total":72,"degraded_ratio":308.40277777777777},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:30.589 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:08:30.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:30.898 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:30.898 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:30.910 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-11T05:08:29.937056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805652,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681708,"statfs":{"total":386547056640,"available":383674068992,"internally_reserved":0,"allocated":1519616,"data_stored":612120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12734,"num_write_kb":-50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.686253"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12626,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263963+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:22.263963+0000","last_peered":"2024-09-11T05:08:22.263963+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:22.263963+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12879,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.475708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:22.475708+0000","last_peered":"2024-09-11T05:08:22.475708+0000","last_clean":"2024-09-11T05:08:22.475708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:22.475708+0000","last_undegraded":"2024-09-11T05:08:22.475708+0000","last_fullsized":"2024-09-11T05:08:22.475708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12282,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.261977+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:22.261977+0000","last_peered":"2024-09-11T05:08:22.261977+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:22.261977+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13946,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257737+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:22.257737+0000","last_peered":"2024-09-11T05:08:22.257737+0000","last_clean":"2024-09-11T05:08:22.257737+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:22.257737+0000","last_undegraded":"2024-09-11T05:08:22.257737+0000","last_fullsized":"2024-09-11T05:08:22.257737+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17134,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.262506+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:22.262506+0000","last_peered":"2024-09-11T05:08:22.262506+0000","last_clean":"2024-09-11T05:08:22.262506+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:22.262506+0000","last_undegraded":"2024-09-11T05:08:22.262506+0000","last_fullsized":"2024-09-11T05:08:22.262506+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12763,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.264715+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:22.264715+0000","last_peered":"2024-09-11T05:08:22.264715+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:22.264715+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17698,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258084+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:22.258084+0000","last_peered":"2024-09-11T05:08:22.258084+0000","last_clean":"2024-09-11T05:08:22.258084+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:22.258084+0000","last_undegraded":"2024-09-11T05:08:22.258084+0000","last_fullsized":"2024-09-11T05:08:22.258084+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262595+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:22.262595+0000","last_peered":"2024-09-11T05:08:22.262595+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:22.262595+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":14137,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:25.069541+0000","last_change":"2024-09-11T05:08:20.776610+0000","last_active":"2024-09-11T05:08:25.069541+0000","last_peered":"2024-09-11T05:08:25.069541+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:25.069541+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12785,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:22.263962+0000","last_peered":"2024-09-11T05:08:22.263962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:22.263962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258195+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:22.258195+0000","last_peered":"2024-09-11T05:08:22.258195+0000","last_clean":"2024-09-11T05:08:22.258195+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:22.258195+0000","last_undegraded":"2024-09-11T05:08:22.258195+0000","last_fullsized":"2024-09-11T05:08:22.258195+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17516,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257742+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:22.257742+0000","last_peered":"2024-09-11T05:08:22.257742+0000","last_clean":"2024-09-11T05:08:22.257742+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:22.257742+0000","last_undegraded":"2024-09-11T05:08:22.257742+0000","last_fullsized":"2024-09-11T05:08:22.257742+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:22.262644+0000","last_peered":"2024-09-11T05:08:22.262644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:22.262644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12544,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263265+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:22.263265+0000","last_peered":"2024-09-11T05:08:22.263265+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:22.263265+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744248,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627592,"statfs":{"total":96636764160,"available":95874654208,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721528,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650312,"statfs":{"total":96636764160,"available":95897919488,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617904,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753936,"statfs":{"total":96636764160,"available":96004030464,"internally_reserved":0,"allocated":425984,"data_stored":199110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721972,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649868,"statfs":{"total":96636764160,"available":95897464832,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:33.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:34.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:34.275 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:34.289 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-11T05:08:33.937775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797332,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2795761,"kb_avail":374690028,"statfs":{"total":386547056640,"available":383682588672,"internally_reserved":0,"allocated":1519616,"data_stored":612120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2862859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12734,"num_write_kb":-50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12626,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263963+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:22.263963+0000","last_peered":"2024-09-11T05:08:22.263963+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:22.263963+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12879,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.475708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:22.475708+0000","last_peered":"2024-09-11T05:08:22.475708+0000","last_clean":"2024-09-11T05:08:22.475708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:22.475708+0000","last_undegraded":"2024-09-11T05:08:22.475708+0000","last_fullsized":"2024-09-11T05:08:22.475708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12282,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.261977+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:22.261977+0000","last_peered":"2024-09-11T05:08:22.261977+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:22.261977+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13946,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257737+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:22.257737+0000","last_peered":"2024-09-11T05:08:22.257737+0000","last_clean":"2024-09-11T05:08:22.257737+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:22.257737+0000","last_undegraded":"2024-09-11T05:08:22.257737+0000","last_fullsized":"2024-09-11T05:08:22.257737+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17134,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.262506+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:22.262506+0000","last_peered":"2024-09-11T05:08:22.262506+0000","last_clean":"2024-09-11T05:08:22.262506+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:22.262506+0000","last_undegraded":"2024-09-11T05:08:22.262506+0000","last_fullsized":"2024-09-11T05:08:22.262506+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12763,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.264715+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:22.264715+0000","last_peered":"2024-09-11T05:08:22.264715+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:22.264715+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17698,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258084+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:22.258084+0000","last_peered":"2024-09-11T05:08:22.258084+0000","last_clean":"2024-09-11T05:08:22.258084+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:22.258084+0000","last_undegraded":"2024-09-11T05:08:22.258084+0000","last_fullsized":"2024-09-11T05:08:22.258084+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262595+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:22.262595+0000","last_peered":"2024-09-11T05:08:22.262595+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:22.262595+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":16019,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:30.071693+0000","last_change":"2024-09-11T05:08:20.776610+0000","last_active":"2024-09-11T05:08:30.071693+0000","last_peered":"2024-09-11T05:08:30.071693+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:30.071693+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":627,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12785,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:22.263962+0000","last_peered":"2024-09-11T05:08:22.263962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:22.263962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258195+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:22.258195+0000","last_peered":"2024-09-11T05:08:22.258195+0000","last_clean":"2024-09-11T05:08:22.258195+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:22.258195+0000","last_undegraded":"2024-09-11T05:08:22.258195+0000","last_fullsized":"2024-09-11T05:08:22.258195+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17516,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257742+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:22.257742+0000","last_peered":"2024-09-11T05:08:22.257742+0000","last_clean":"2024-09-11T05:08:22.257742+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:22.257742+0000","last_undegraded":"2024-09-11T05:08:22.257742+0000","last_fullsized":"2024-09-11T05:08:22.257742+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:22.262644+0000","last_peered":"2024-09-11T05:08:22.262644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:22.262644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12544,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263265+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:22.263265+0000","last_peered":"2024-09-11T05:08:22.263265+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:22.263265+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744248,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627592,"statfs":{"total":96636764160,"available":95874654208,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721528,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650312,"statfs":{"total":96636764160,"available":95897919488,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609584,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762256,"statfs":{"total":96636764160,"available":96012550144,"internally_reserved":0,"allocated":425984,"data_stored":199110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623767250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721972,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649868,"statfs":{"total":96636764160,"available":95897464832,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:34.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:34.632 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:34.647 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-11T05:08:33.937775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797332,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2795761,"kb_avail":374690028,"statfs":{"total":386547056640,"available":383682588672,"internally_reserved":0,"allocated":1519616,"data_stored":612120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2862859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12734,"num_write_kb":-50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12626,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263963+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:22.263963+0000","last_peered":"2024-09-11T05:08:22.263963+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:22.263963+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12879,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.475708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:22.475708+0000","last_peered":"2024-09-11T05:08:22.475708+0000","last_clean":"2024-09-11T05:08:22.475708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:22.475708+0000","last_undegraded":"2024-09-11T05:08:22.475708+0000","last_fullsized":"2024-09-11T05:08:22.475708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12282,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.261977+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:22.261977+0000","last_peered":"2024-09-11T05:08:22.261977+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:22.261977+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13946,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257737+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:22.257737+0000","last_peered":"2024-09-11T05:08:22.257737+0000","last_clean":"2024-09-11T05:08:22.257737+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:22.257737+0000","last_undegraded":"2024-09-11T05:08:22.257737+0000","last_fullsized":"2024-09-11T05:08:22.257737+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17134,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.262506+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:22.262506+0000","last_peered":"2024-09-11T05:08:22.262506+0000","last_clean":"2024-09-11T05:08:22.262506+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:22.262506+0000","last_undegraded":"2024-09-11T05:08:22.262506+0000","last_fullsized":"2024-09-11T05:08:22.262506+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12763,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.264715+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:22.264715+0000","last_peered":"2024-09-11T05:08:22.264715+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:22.264715+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17698,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258084+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:22.258084+0000","last_peered":"2024-09-11T05:08:22.258084+0000","last_clean":"2024-09-11T05:08:22.258084+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:22.258084+0000","last_undegraded":"2024-09-11T05:08:22.258084+0000","last_fullsized":"2024-09-11T05:08:22.258084+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262595+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:22.262595+0000","last_peered":"2024-09-11T05:08:22.262595+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:22.262595+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":16019,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:30.071693+0000","last_change":"2024-09-11T05:08:20.776610+0000","last_active":"2024-09-11T05:08:30.071693+0000","last_peered":"2024-09-11T05:08:30.071693+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:30.071693+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":627,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12785,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:22.263962+0000","last_peered":"2024-09-11T05:08:22.263962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:22.263962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258195+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:22.258195+0000","last_peered":"2024-09-11T05:08:22.258195+0000","last_clean":"2024-09-11T05:08:22.258195+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:22.258195+0000","last_undegraded":"2024-09-11T05:08:22.258195+0000","last_fullsized":"2024-09-11T05:08:22.258195+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17516,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257742+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:22.257742+0000","last_peered":"2024-09-11T05:08:22.257742+0000","last_clean":"2024-09-11T05:08:22.257742+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:22.257742+0000","last_undegraded":"2024-09-11T05:08:22.257742+0000","last_fullsized":"2024-09-11T05:08:22.257742+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:22.262644+0000","last_peered":"2024-09-11T05:08:22.262644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:22.262644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12544,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263265+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:22.263265+0000","last_peered":"2024-09-11T05:08:22.263265+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:22.263265+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744248,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627592,"statfs":{"total":96636764160,"available":95874654208,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721528,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650312,"statfs":{"total":96636764160,"available":95897919488,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609584,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762256,"statfs":{"total":96636764160,"available":96012550144,"internally_reserved":0,"allocated":425984,"data_stored":199110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623767250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721972,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649868,"statfs":{"total":96636764160,"available":95897464832,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12626, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.263963+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:22.263963+0000', 'last_peered': '2024-09-11T05:08:22.263963+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:22.263963+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12282, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.261977+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:22.261977+0000', 'last_peered': '2024-09-11T05:08:22.261977+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:22.261977+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12763, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.264715+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:22.264715+0000', 'last_peered': '2024-09-11T05:08:22.264715+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:22.264715+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 12840, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.258452+0000', 'last_change': '2024-09-11T04:56:20.631096+0000', 'last_active': '2024-09-11T05:08:22.258452+0000', 'last_peered': '2024-09-11T05:08:22.258452+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:08:22.258452+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2309, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2305}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:34.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12925, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.262595+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:22.262595+0000', 'last_peered': '2024-09-11T05:08:22.262595+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:22.262595+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 16019, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:30.071693+0000', 'last_change': '2024-09-11T05:08:20.776610+0000', 'last_active': '2024-09-11T05:08:30.071693+0000', 'last_peered': '2024-09-11T05:08:30.071693+0000', 'last_clean': '2024-09-11T04:54:59.713168+0000', 'last_became_active': '2024-09-11T04:56:20.584675+0000', 'last_became_peered': '2024-09-11T04:56:20.584675+0000', 'last_unstale': '2024-09-11T05:08:30.071693+0000', 'last_undegraded': '2024-09-11T04:56:20.270199+0000', 'last_fullsized': '2024-09-11T04:56:20.225599+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 627, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 50102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1695, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -625}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12785, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.263962+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:22.263962+0000', 'last_peered': '2024-09-11T05:08:22.263962+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:22.263962+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12714, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.262644+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:08:22.262644+0000', 'last_peered': '2024-09-11T05:08:22.262644+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:22.262644+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12544, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.263265+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:22.263265+0000', 'last_peered': '2024-09-11T05:08:22.263265+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:22.263265+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:34.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12912, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:22.259413+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:22.259413+0000', 'last_peered': '2024-09-11T05:08:22.259413+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:22.259413+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:34.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:35.057 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21264/72 objects degraded (29533.333%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":842,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":121248,"bytes_used":2864467968,"bytes_avail":383682588672,"bytes_total":386547056640,"degraded_objects":21264,"degraded_total":72,"degraded_ratio":295.33333333333331},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:35.070 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:08:35.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:35.394 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:35.395 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:35.408 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-11T05:08:33.937775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797332,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2795761,"kb_avail":374690028,"statfs":{"total":386547056640,"available":383682588672,"internally_reserved":0,"allocated":1519616,"data_stored":612120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2862859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12734,"num_write_kb":-50948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1342,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12626,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263963+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:22.263963+0000","last_peered":"2024-09-11T05:08:22.263963+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:22.263963+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12879,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.475708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:22.475708+0000","last_peered":"2024-09-11T05:08:22.475708+0000","last_clean":"2024-09-11T05:08:22.475708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:22.475708+0000","last_undegraded":"2024-09-11T05:08:22.475708+0000","last_fullsized":"2024-09-11T05:08:22.475708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12282,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.261977+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:22.261977+0000","last_peered":"2024-09-11T05:08:22.261977+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:22.261977+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13946,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257737+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:22.257737+0000","last_peered":"2024-09-11T05:08:22.257737+0000","last_clean":"2024-09-11T05:08:22.257737+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:22.257737+0000","last_undegraded":"2024-09-11T05:08:22.257737+0000","last_fullsized":"2024-09-11T05:08:22.257737+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17134,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.262506+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:22.262506+0000","last_peered":"2024-09-11T05:08:22.262506+0000","last_clean":"2024-09-11T05:08:22.262506+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:22.262506+0000","last_undegraded":"2024-09-11T05:08:22.262506+0000","last_fullsized":"2024-09-11T05:08:22.262506+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12763,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.264715+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:22.264715+0000","last_peered":"2024-09-11T05:08:22.264715+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:22.264715+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":12840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.258452+0000","last_change":"2024-09-11T04:56:20.631096+0000","last_active":"2024-09-11T05:08:22.258452+0000","last_peered":"2024-09-11T05:08:22.258452+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:22.258452+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17698,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258084+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:22.258084+0000","last_peered":"2024-09-11T05:08:22.258084+0000","last_clean":"2024-09-11T05:08:22.258084+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:22.258084+0000","last_undegraded":"2024-09-11T05:08:22.258084+0000","last_fullsized":"2024-09-11T05:08:22.258084+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262595+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:22.262595+0000","last_peered":"2024-09-11T05:08:22.262595+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:22.262595+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":16019,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:30.071693+0000","last_change":"2024-09-11T05:08:20.776610+0000","last_active":"2024-09-11T05:08:30.071693+0000","last_peered":"2024-09-11T05:08:30.071693+0000","last_clean":"2024-09-11T04:54:59.713168+0000","last_became_active":"2024-09-11T04:56:20.584675+0000","last_became_peered":"2024-09-11T04:56:20.584675+0000","last_unstale":"2024-09-11T05:08:30.071693+0000","last_undegraded":"2024-09-11T04:56:20.270199+0000","last_fullsized":"2024-09-11T04:56:20.225599+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":627,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12785,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263962+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:22.263962+0000","last_peered":"2024-09-11T05:08:22.263962+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:22.263962+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.258195+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:22.258195+0000","last_peered":"2024-09-11T05:08:22.258195+0000","last_clean":"2024-09-11T05:08:22.258195+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:22.258195+0000","last_undegraded":"2024-09-11T05:08:22.258195+0000","last_fullsized":"2024-09-11T05:08:22.258195+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17516,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-11T05:08:22.257742+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:22.257742+0000","last_peered":"2024-09-11T05:08:22.257742+0000","last_clean":"2024-09-11T05:08:22.257742+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:22.257742+0000","last_undegraded":"2024-09-11T05:08:22.257742+0000","last_fullsized":"2024-09-11T05:08:22.257742+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.262644+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:22.262644+0000","last_peered":"2024-09-11T05:08:22.262644+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:22.262644+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12544,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.263265+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:22.263265+0000","last_peered":"2024-09-11T05:08:22.263265+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:22.263265+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12912,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:22.259413+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:22.259413+0000","last_peered":"2024-09-11T05:08:22.259413+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:22.259413+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121248,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136353,"num_write_kb":503107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744248,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627592,"statfs":{"total":96636764160,"available":95874654208,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721528,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650312,"statfs":{"total":96636764160,"available":95897919488,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609584,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762256,"statfs":{"total":96636764160,"available":96012550144,"internally_reserved":0,"allocated":425984,"data_stored":199110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623767250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721972,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649868,"statfs":{"total":96636764160,"available":95897464832,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:38.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:38.738 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:38.750 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-11T05:08:37.938453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805716,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681644,"statfs":{"total":386547056640,"available":383674003456,"internally_reserved":0,"allocated":1576960,"data_stored":631052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12524,"num_write_kb":-50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001397"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12630,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913909+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:33.913909+0000","last_peered":"2024-09-11T05:08:33.913909+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:33.913909+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12883,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920116+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:33.920116+0000","last_peered":"2024-09-11T05:08:33.920116+0000","last_clean":"2024-09-11T05:08:33.920116+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:33.920116+0000","last_undegraded":"2024-09-11T05:08:33.920116+0000","last_fullsized":"2024-09-11T05:08:33.920116+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12287,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.917950+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:33.917950+0000","last_peered":"2024-09-11T05:08:33.917950+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:33.917950+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13950,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918448+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:33.918448+0000","last_peered":"2024-09-11T05:08:33.918448+0000","last_clean":"2024-09-11T05:08:33.918448+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:33.918448+0000","last_undegraded":"2024-09-11T05:08:33.918448+0000","last_fullsized":"2024-09-11T05:08:33.918448+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17138,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.913597+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:33.913597+0000","last_peered":"2024-09-11T05:08:33.913597+0000","last_clean":"2024-09-11T05:08:33.913597+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:33.913597+0000","last_undegraded":"2024-09-11T05:08:33.913597+0000","last_fullsized":"2024-09-11T05:08:33.913597+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12767,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.915425+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:33.915425+0000","last_peered":"2024-09-11T05:08:33.915425+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:33.915425+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":14567,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:37.407306+0000","last_change":"2024-09-11T05:08:32.586339+0000","last_active":"2024-09-11T05:08:37.407306+0000","last_peered":"2024-09-11T05:08:37.407306+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:37.407306+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17702,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.919749+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:33.919749+0000","last_peered":"2024-09-11T05:08:33.919749+0000","last_clean":"2024-09-11T05:08:33.919749+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:33.919749+0000","last_undegraded":"2024-09-11T05:08:33.919749+0000","last_fullsized":"2024-09-11T05:08:33.919749+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12929,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.914329+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:33.914329+0000","last_peered":"2024-09-11T05:08:33.914329+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:33.914329+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:34.206929+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:34.206929+0000","last_peered":"2024-09-11T05:08:34.206929+0000","last_clean":"2024-09-11T05:08:34.206929+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:34.206929+0000","last_undegraded":"2024-09-11T05:08:34.206929+0000","last_fullsized":"2024-09-11T05:08:34.206929+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12789,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913854+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:33.913854+0000","last_peered":"2024-09-11T05:08:33.913854+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:33.913854+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920029+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:33.920029+0000","last_peered":"2024-09-11T05:08:33.920029+0000","last_clean":"2024-09-11T05:08:33.920029+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:33.920029+0000","last_undegraded":"2024-09-11T05:08:33.920029+0000","last_fullsized":"2024-09-11T05:08:33.920029+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17520,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918539+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:33.918539+0000","last_peered":"2024-09-11T05:08:33.918539+0000","last_clean":"2024-09-11T05:08:33.918539+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:33.918539+0000","last_undegraded":"2024-09-11T05:08:33.918539+0000","last_fullsized":"2024-09-11T05:08:33.918539+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12718,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913631+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:33.913631+0000","last_peered":"2024-09-11T05:08:33.913631+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:33.913631+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12548,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.916763+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:33.916763+0000","last_peered":"2024-09-11T05:08:33.916763+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:33.916763+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12916,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.919414+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:33.919414+0000","last_peered":"2024-09-11T05:08:33.919414+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:33.919414+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744264,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627576,"statfs":{"total":96636764160,"available":95874637824,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617912,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753928,"statfs":{"total":96636764160,"available":96004022272,"internally_reserved":0,"allocated":434176,"data_stored":197699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:38.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:39.062 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:39.062 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-11T05:08:37.938453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805716,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681644,"statfs":{"total":386547056640,"available":383674003456,"internally_reserved":0,"allocated":1576960,"data_stored":631052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12524,"num_write_kb":-50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001397"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12630,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913909+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:33.913909+0000","last_peered":"2024-09-11T05:08:33.913909+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:33.913909+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12883,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920116+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:33.920116+0000","last_peered":"2024-09-11T05:08:33.920116+0000","last_clean":"2024-09-11T05:08:33.920116+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:33.920116+0000","last_undegraded":"2024-09-11T05:08:33.920116+0000","last_fullsized":"2024-09-11T05:08:33.920116+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12287,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.917950+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:33.917950+0000","last_peered":"2024-09-11T05:08:33.917950+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:33.917950+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13950,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918448+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:33.918448+0000","last_peered":"2024-09-11T05:08:33.918448+0000","last_clean":"2024-09-11T05:08:33.918448+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:33.918448+0000","last_undegraded":"2024-09-11T05:08:33.918448+0000","last_fullsized":"2024-09-11T05:08:33.918448+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17138,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.913597+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:33.913597+0000","last_peered":"2024-09-11T05:08:33.913597+0000","last_clean":"2024-09-11T05:08:33.913597+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:33.913597+0000","last_undegraded":"2024-09-11T05:08:33.913597+0000","last_fullsized":"2024-09-11T05:08:33.913597+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12767,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.915425+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:33.915425+0000","last_peered":"2024-09-11T05:08:33.915425+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:33.915425+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":14567,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:37.407306+0000","last_change":"2024-09-11T05:08:32.586339+0000","last_active":"2024-09-11T05:08:37.407306+0000","last_peered":"2024-09-11T05:08:37.407306+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:37.407306+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17702,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.919749+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:33.919749+0000","last_peered":"2024-09-11T05:08:33.919749+0000","last_clean":"2024-09-11T05:08:33.919749+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:33.919749+0000","last_undegraded":"2024-09-11T05:08:33.919749+0000","last_fullsized":"2024-09-11T05:08:33.919749+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12929,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.914329+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:33.914329+0000","last_peered":"2024-09-11T05:08:33.914329+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:33.914329+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:34.206929+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:34.206929+0000","last_peered":"2024-09-11T05:08:34.206929+0000","last_clean":"2024-09-11T05:08:34.206929+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:34.206929+0000","last_undegraded":"2024-09-11T05:08:34.206929+0000","last_fullsized":"2024-09-11T05:08:34.206929+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12789,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913854+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:33.913854+0000","last_peered":"2024-09-11T05:08:33.913854+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:33.913854+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920029+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:33.920029+0000","last_peered":"2024-09-11T05:08:33.920029+0000","last_clean":"2024-09-11T05:08:33.920029+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:33.920029+0000","last_undegraded":"2024-09-11T05:08:33.920029+0000","last_fullsized":"2024-09-11T05:08:33.920029+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17520,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918539+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:33.918539+0000","last_peered":"2024-09-11T05:08:33.918539+0000","last_clean":"2024-09-11T05:08:33.918539+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:33.918539+0000","last_undegraded":"2024-09-11T05:08:33.918539+0000","last_fullsized":"2024-09-11T05:08:33.918539+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12718,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913631+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:33.913631+0000","last_peered":"2024-09-11T05:08:33.913631+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:33.913631+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12548,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.916763+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:33.916763+0000","last_peered":"2024-09-11T05:08:33.916763+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:33.916763+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12916,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.919414+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:33.919414+0000","last_peered":"2024-09-11T05:08:33.919414+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:33.919414+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744264,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627576,"statfs":{"total":96636764160,"available":95874637824,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617912,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753928,"statfs":{"total":96636764160,"available":96004022272,"internally_reserved":0,"allocated":434176,"data_stored":197699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:39.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:39.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12630, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.913909+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:33.913909+0000', 'last_peered': '2024-09-11T05:08:33.913909+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:33.913909+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12287, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.917950+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:33.917950+0000', 'last_peered': '2024-09-11T05:08:33.917950+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:33.917950+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12767, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.915425+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:33.915425+0000', 'last_peered': '2024-09-11T05:08:33.915425+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:33.915425+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 14567, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:37.407306+0000', 'last_change': '2024-09-11T05:08:32.586339+0000', 'last_active': '2024-09-11T05:08:37.407306+0000', 'last_peered': '2024-09-11T05:08:37.407306+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:08:37.407306+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1448, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 873, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1446}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12929, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.914329+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:33.914329+0000', 'last_peered': '2024-09-11T05:08:33.914329+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:33.914329+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:39.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:39.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12789, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.913854+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:33.913854+0000', 'last_peered': '2024-09-11T05:08:33.913854+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:33.913854+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:39.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:39.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12718, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.913631+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:08:33.913631+0000', 'last_peered': '2024-09-11T05:08:33.913631+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:33.913631+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:39.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:39.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12548, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.916763+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:33.916763+0000', 'last_peered': '2024-09-11T05:08:33.916763+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:33.916763+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:39.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:39.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12916, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.919414+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:33.919414+0000', 'last_peered': '2024-09-11T05:08:33.919414+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:33.919414+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:39.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:39.490 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:39.491 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19776/64 objects degraded (30900.000%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":846,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":107776,"bytes_used":2873053184,"bytes_avail":383674003456,"bytes_total":386547056640,"degraded_objects":19776,"degraded_total":64,"degraded_ratio":309,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:39.505 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:08:39.505 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:39.815 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:39.816 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:39.828 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-11T05:08:37.938453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805716,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681644,"statfs":{"total":386547056640,"available":383674003456,"internally_reserved":0,"allocated":1576960,"data_stored":631052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12524,"num_write_kb":-50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001397"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12630,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913909+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:33.913909+0000","last_peered":"2024-09-11T05:08:33.913909+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:33.913909+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12883,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920116+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:33.920116+0000","last_peered":"2024-09-11T05:08:33.920116+0000","last_clean":"2024-09-11T05:08:33.920116+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:33.920116+0000","last_undegraded":"2024-09-11T05:08:33.920116+0000","last_fullsized":"2024-09-11T05:08:33.920116+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12287,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.917950+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:33.917950+0000","last_peered":"2024-09-11T05:08:33.917950+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:33.917950+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13950,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918448+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:33.918448+0000","last_peered":"2024-09-11T05:08:33.918448+0000","last_clean":"2024-09-11T05:08:33.918448+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:33.918448+0000","last_undegraded":"2024-09-11T05:08:33.918448+0000","last_fullsized":"2024-09-11T05:08:33.918448+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17138,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.913597+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:33.913597+0000","last_peered":"2024-09-11T05:08:33.913597+0000","last_clean":"2024-09-11T05:08:33.913597+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:33.913597+0000","last_undegraded":"2024-09-11T05:08:33.913597+0000","last_fullsized":"2024-09-11T05:08:33.913597+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12767,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.915425+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:33.915425+0000","last_peered":"2024-09-11T05:08:33.915425+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:33.915425+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":14567,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:37.407306+0000","last_change":"2024-09-11T05:08:32.586339+0000","last_active":"2024-09-11T05:08:37.407306+0000","last_peered":"2024-09-11T05:08:37.407306+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:37.407306+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17702,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.919749+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:33.919749+0000","last_peered":"2024-09-11T05:08:33.919749+0000","last_clean":"2024-09-11T05:08:33.919749+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:33.919749+0000","last_undegraded":"2024-09-11T05:08:33.919749+0000","last_fullsized":"2024-09-11T05:08:33.919749+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12929,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.914329+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:33.914329+0000","last_peered":"2024-09-11T05:08:33.914329+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:33.914329+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:34.206929+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:34.206929+0000","last_peered":"2024-09-11T05:08:34.206929+0000","last_clean":"2024-09-11T05:08:34.206929+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:34.206929+0000","last_undegraded":"2024-09-11T05:08:34.206929+0000","last_fullsized":"2024-09-11T05:08:34.206929+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12789,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913854+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:33.913854+0000","last_peered":"2024-09-11T05:08:33.913854+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:33.913854+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920029+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:33.920029+0000","last_peered":"2024-09-11T05:08:33.920029+0000","last_clean":"2024-09-11T05:08:33.920029+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:33.920029+0000","last_undegraded":"2024-09-11T05:08:33.920029+0000","last_fullsized":"2024-09-11T05:08:33.920029+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17520,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918539+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:33.918539+0000","last_peered":"2024-09-11T05:08:33.918539+0000","last_clean":"2024-09-11T05:08:33.918539+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:33.918539+0000","last_undegraded":"2024-09-11T05:08:33.918539+0000","last_fullsized":"2024-09-11T05:08:33.918539+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12718,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913631+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:33.913631+0000","last_peered":"2024-09-11T05:08:33.913631+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:33.913631+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12548,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.916763+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:33.916763+0000","last_peered":"2024-09-11T05:08:33.916763+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:33.916763+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12916,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.919414+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:33.919414+0000","last_peered":"2024-09-11T05:08:33.919414+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:33.919414+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744264,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627576,"statfs":{"total":96636764160,"available":95874637824,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617912,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753928,"statfs":{"total":96636764160,"available":96004022272,"internally_reserved":0,"allocated":434176,"data_stored":197699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:42.830 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:43.144 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:43.156 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-11T05:08:41.939206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805716,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681644,"statfs":{"total":386547056640,"available":383674003456,"internally_reserved":0,"allocated":1576960,"data_stored":631052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12524,"num_write_kb":-50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-834,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.026774"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12630,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913909+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:33.913909+0000","last_peered":"2024-09-11T05:08:33.913909+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:33.913909+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12883,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920116+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:33.920116+0000","last_peered":"2024-09-11T05:08:33.920116+0000","last_clean":"2024-09-11T05:08:33.920116+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:33.920116+0000","last_undegraded":"2024-09-11T05:08:33.920116+0000","last_fullsized":"2024-09-11T05:08:33.920116+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12287,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.917950+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:33.917950+0000","last_peered":"2024-09-11T05:08:33.917950+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:33.917950+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13950,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918448+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:33.918448+0000","last_peered":"2024-09-11T05:08:33.918448+0000","last_clean":"2024-09-11T05:08:33.918448+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:33.918448+0000","last_undegraded":"2024-09-11T05:08:33.918448+0000","last_fullsized":"2024-09-11T05:08:33.918448+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17138,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.913597+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:33.913597+0000","last_peered":"2024-09-11T05:08:33.913597+0000","last_clean":"2024-09-11T05:08:33.913597+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:33.913597+0000","last_undegraded":"2024-09-11T05:08:33.913597+0000","last_fullsized":"2024-09-11T05:08:33.913597+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12767,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.915425+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:33.915425+0000","last_peered":"2024-09-11T05:08:33.915425+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:33.915425+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":14567,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:37.407306+0000","last_change":"2024-09-11T05:08:32.586339+0000","last_active":"2024-09-11T05:08:37.407306+0000","last_peered":"2024-09-11T05:08:37.407306+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:37.407306+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17702,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.919749+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:33.919749+0000","last_peered":"2024-09-11T05:08:33.919749+0000","last_clean":"2024-09-11T05:08:33.919749+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:33.919749+0000","last_undegraded":"2024-09-11T05:08:33.919749+0000","last_fullsized":"2024-09-11T05:08:33.919749+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12929,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.914329+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:33.914329+0000","last_peered":"2024-09-11T05:08:33.914329+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:33.914329+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:34.206929+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:34.206929+0000","last_peered":"2024-09-11T05:08:34.206929+0000","last_clean":"2024-09-11T05:08:34.206929+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:34.206929+0000","last_undegraded":"2024-09-11T05:08:34.206929+0000","last_fullsized":"2024-09-11T05:08:34.206929+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12789,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913854+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:33.913854+0000","last_peered":"2024-09-11T05:08:33.913854+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:33.913854+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920029+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:33.920029+0000","last_peered":"2024-09-11T05:08:33.920029+0000","last_clean":"2024-09-11T05:08:33.920029+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:33.920029+0000","last_undegraded":"2024-09-11T05:08:33.920029+0000","last_fullsized":"2024-09-11T05:08:33.920029+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17520,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918539+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:33.918539+0000","last_peered":"2024-09-11T05:08:33.918539+0000","last_clean":"2024-09-11T05:08:33.918539+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:33.918539+0000","last_undegraded":"2024-09-11T05:08:33.918539+0000","last_fullsized":"2024-09-11T05:08:33.918539+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12718,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913631+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:33.913631+0000","last_peered":"2024-09-11T05:08:33.913631+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:33.913631+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12548,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.916763+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:33.916763+0000","last_peered":"2024-09-11T05:08:33.916763+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:33.916763+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12916,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.919414+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:33.919414+0000","last_peered":"2024-09-11T05:08:33.919414+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:33.919414+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744264,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627576,"statfs":{"total":96636764160,"available":95874637824,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617912,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753928,"statfs":{"total":96636764160,"available":96004022272,"internally_reserved":0,"allocated":434176,"data_stored":197699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:43.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:43.476 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:43.488 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-11T05:08:41.939206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805716,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681644,"statfs":{"total":386547056640,"available":383674003456,"internally_reserved":0,"allocated":1576960,"data_stored":631052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12524,"num_write_kb":-50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-834,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.026774"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12630,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913909+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:33.913909+0000","last_peered":"2024-09-11T05:08:33.913909+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:33.913909+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12883,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920116+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:33.920116+0000","last_peered":"2024-09-11T05:08:33.920116+0000","last_clean":"2024-09-11T05:08:33.920116+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:33.920116+0000","last_undegraded":"2024-09-11T05:08:33.920116+0000","last_fullsized":"2024-09-11T05:08:33.920116+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12287,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.917950+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:33.917950+0000","last_peered":"2024-09-11T05:08:33.917950+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:33.917950+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13950,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918448+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:33.918448+0000","last_peered":"2024-09-11T05:08:33.918448+0000","last_clean":"2024-09-11T05:08:33.918448+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:33.918448+0000","last_undegraded":"2024-09-11T05:08:33.918448+0000","last_fullsized":"2024-09-11T05:08:33.918448+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17138,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.913597+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:33.913597+0000","last_peered":"2024-09-11T05:08:33.913597+0000","last_clean":"2024-09-11T05:08:33.913597+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:33.913597+0000","last_undegraded":"2024-09-11T05:08:33.913597+0000","last_fullsized":"2024-09-11T05:08:33.913597+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12767,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.915425+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:33.915425+0000","last_peered":"2024-09-11T05:08:33.915425+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:33.915425+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":14567,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:37.407306+0000","last_change":"2024-09-11T05:08:32.586339+0000","last_active":"2024-09-11T05:08:37.407306+0000","last_peered":"2024-09-11T05:08:37.407306+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:37.407306+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17702,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.919749+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:33.919749+0000","last_peered":"2024-09-11T05:08:33.919749+0000","last_clean":"2024-09-11T05:08:33.919749+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:33.919749+0000","last_undegraded":"2024-09-11T05:08:33.919749+0000","last_fullsized":"2024-09-11T05:08:33.919749+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12929,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.914329+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:33.914329+0000","last_peered":"2024-09-11T05:08:33.914329+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:33.914329+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:34.206929+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:34.206929+0000","last_peered":"2024-09-11T05:08:34.206929+0000","last_clean":"2024-09-11T05:08:34.206929+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:34.206929+0000","last_undegraded":"2024-09-11T05:08:34.206929+0000","last_fullsized":"2024-09-11T05:08:34.206929+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12789,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913854+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:33.913854+0000","last_peered":"2024-09-11T05:08:33.913854+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:33.913854+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920029+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:33.920029+0000","last_peered":"2024-09-11T05:08:33.920029+0000","last_clean":"2024-09-11T05:08:33.920029+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:33.920029+0000","last_undegraded":"2024-09-11T05:08:33.920029+0000","last_fullsized":"2024-09-11T05:08:33.920029+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17520,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918539+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:33.918539+0000","last_peered":"2024-09-11T05:08:33.918539+0000","last_clean":"2024-09-11T05:08:33.918539+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:33.918539+0000","last_undegraded":"2024-09-11T05:08:33.918539+0000","last_fullsized":"2024-09-11T05:08:33.918539+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12718,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913631+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:33.913631+0000","last_peered":"2024-09-11T05:08:33.913631+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:33.913631+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12548,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.916763+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:33.916763+0000","last_peered":"2024-09-11T05:08:33.916763+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:33.916763+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12916,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.919414+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:33.919414+0000","last_peered":"2024-09-11T05:08:33.919414+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:33.919414+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744264,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627576,"statfs":{"total":96636764160,"available":95874637824,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617912,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753928,"statfs":{"total":96636764160,"available":96004022272,"internally_reserved":0,"allocated":434176,"data_stored":197699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12630, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.913909+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:33.913909+0000', 'last_peered': '2024-09-11T05:08:33.913909+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:33.913909+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12287, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.917950+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:33.917950+0000', 'last_peered': '2024-09-11T05:08:33.917950+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:33.917950+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12767, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.915425+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:33.915425+0000', 'last_peered': '2024-09-11T05:08:33.915425+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:33.915425+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12728", 'reported_seq': 14567, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:37.407306+0000', 'last_change': '2024-09-11T05:08:32.586339+0000', 'last_active': '2024-09-11T05:08:37.407306+0000', 'last_peered': '2024-09-11T05:08:37.407306+0000', 'last_clean': '2024-09-11T04:55:08.774376+0000', 'last_became_active': '2024-09-11T04:56:20.617387+0000', 'last_became_peered': '2024-09-11T04:56:20.617387+0000', 'last_unstale': '2024-09-11T05:08:37.407306+0000', 'last_undegraded': '2024-09-11T04:56:20.287616+0000', 'last_fullsized': '2024-09-11T04:56:20.222916+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1448, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12726, 'num_write_kb': 50908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 873, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1446}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:43.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12929, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.914329+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:33.914329+0000', 'last_peered': '2024-09-11T05:08:33.914329+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:33.914329+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12789, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.913854+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:33.913854+0000', 'last_peered': '2024-09-11T05:08:33.913854+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:33.913854+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12718, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.913631+0000', 'last_change': '2024-09-11T04:56:20.593644+0000', 'last_active': '2024-09-11T05:08:33.913631+0000', 'last_peered': '2024-09-11T05:08:33.913631+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:33.913631+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12548, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.916763+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:33.916763+0000', 'last_peered': '2024-09-11T05:08:33.916763+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:33.916763+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:43.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12916, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:33.919414+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:33.919414+0000', 'last_peered': '2024-09-11T05:08:33.919414+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:33.919414+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:43.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:43.918 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:43.918 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19776/64 objects degraded (30900.000%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":850,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":107776,"bytes_used":2873053184,"bytes_avail":383674003456,"bytes_total":386547056640,"degraded_objects":19776,"degraded_total":64,"degraded_ratio":309},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:43.931 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:08:43.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:44.263 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:44.277 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-11T05:08:43.939564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2089,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797396,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2795761,"kb_avail":374689964,"statfs":{"total":386547056640,"available":383682523136,"internally_reserved":0,"allocated":1576960,"data_stored":631052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2862859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12524,"num_write_kb":-50102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.026500"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12630,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913909+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:33.913909+0000","last_peered":"2024-09-11T05:08:33.913909+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:33.913909+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12883,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920116+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:33.920116+0000","last_peered":"2024-09-11T05:08:33.920116+0000","last_clean":"2024-09-11T05:08:33.920116+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:33.920116+0000","last_undegraded":"2024-09-11T05:08:33.920116+0000","last_fullsized":"2024-09-11T05:08:33.920116+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12287,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.917950+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:33.917950+0000","last_peered":"2024-09-11T05:08:33.917950+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:33.917950+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13950,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918448+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:33.918448+0000","last_peered":"2024-09-11T05:08:33.918448+0000","last_clean":"2024-09-11T05:08:33.918448+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:33.918448+0000","last_undegraded":"2024-09-11T05:08:33.918448+0000","last_fullsized":"2024-09-11T05:08:33.918448+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17138,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.913597+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:33.913597+0000","last_peered":"2024-09-11T05:08:33.913597+0000","last_clean":"2024-09-11T05:08:33.913597+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:33.913597+0000","last_undegraded":"2024-09-11T05:08:33.913597+0000","last_fullsized":"2024-09-11T05:08:33.913597+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12767,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.915425+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:33.915425+0000","last_peered":"2024-09-11T05:08:33.915425+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:33.915425+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":16631,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:42.411226+0000","last_change":"2024-09-11T05:08:32.586339+0000","last_active":"2024-09-11T05:08:42.411226+0000","last_peered":"2024-09-11T05:08:42.411226+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:42.411226+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1905,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-414}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17702,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.919749+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:33.919749+0000","last_peered":"2024-09-11T05:08:33.919749+0000","last_clean":"2024-09-11T05:08:33.919749+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:33.919749+0000","last_undegraded":"2024-09-11T05:08:33.919749+0000","last_fullsized":"2024-09-11T05:08:33.919749+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12929,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.914329+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:33.914329+0000","last_peered":"2024-09-11T05:08:33.914329+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:33.914329+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:34.206929+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:34.206929+0000","last_peered":"2024-09-11T05:08:34.206929+0000","last_clean":"2024-09-11T05:08:34.206929+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:34.206929+0000","last_undegraded":"2024-09-11T05:08:34.206929+0000","last_fullsized":"2024-09-11T05:08:34.206929+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12789,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913854+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:33.913854+0000","last_peered":"2024-09-11T05:08:33.913854+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:33.913854+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920029+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:33.920029+0000","last_peered":"2024-09-11T05:08:33.920029+0000","last_clean":"2024-09-11T05:08:33.920029+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:33.920029+0000","last_undegraded":"2024-09-11T05:08:33.920029+0000","last_fullsized":"2024-09-11T05:08:33.920029+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17520,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918539+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:33.918539+0000","last_peered":"2024-09-11T05:08:33.918539+0000","last_clean":"2024-09-11T05:08:33.918539+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:33.918539+0000","last_undegraded":"2024-09-11T05:08:33.918539+0000","last_fullsized":"2024-09-11T05:08:33.918539+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12718,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.913631+0000","last_change":"2024-09-11T04:56:20.593644+0000","last_active":"2024-09-11T05:08:33.913631+0000","last_peered":"2024-09-11T05:08:33.913631+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:33.913631+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2262}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12548,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.916763+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:33.916763+0000","last_peered":"2024-09-11T05:08:33.916763+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:33.916763+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12917,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.920639+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:33.920639+0000","last_peered":"2024-09-11T05:08:33.920639+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:33.920639+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2089,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744264,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627576,"statfs":{"total":96636764160,"available":95874637824,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609592,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762248,"statfs":{"total":96636764160,"available":96012541952,"internally_reserved":0,"allocated":434176,"data_stored":197699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623767250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:47.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:47.584 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:47.584 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:47.596 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-11T05:08:46.027278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797404,"kb_used_data":1548,"kb_used_omap":14,"kb_used_meta":2795761,"kb_avail":374689956,"statfs":{"total":386547056640,"available":383682514944,"internally_reserved":0,"allocated":1585152,"data_stored":634416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2862859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1209,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.088825"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12632,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012902+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:45.012902+0000","last_peered":"2024-09-11T05:08:45.012902+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:45.012902+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12883,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920116+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:33.920116+0000","last_peered":"2024-09-11T05:08:33.920116+0000","last_clean":"2024-09-11T05:08:33.920116+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:33.920116+0000","last_undegraded":"2024-09-11T05:08:33.920116+0000","last_fullsized":"2024-09-11T05:08:33.920116+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12289,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.010274+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:45.010274+0000","last_peered":"2024-09-11T05:08:45.010274+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:45.010274+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13950,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918448+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:33.918448+0000","last_peered":"2024-09-11T05:08:33.918448+0000","last_clean":"2024-09-11T05:08:33.918448+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:33.918448+0000","last_undegraded":"2024-09-11T05:08:33.918448+0000","last_fullsized":"2024-09-11T05:08:33.918448+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17140,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-11T05:08:45.011234+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:45.011234+0000","last_peered":"2024-09-11T05:08:45.011234+0000","last_clean":"2024-09-11T05:08:45.011234+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:45.011234+0000","last_undegraded":"2024-09-11T05:08:45.011234+0000","last_fullsized":"2024-09-11T05:08:45.011234+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12769,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012434+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:45.012434+0000","last_peered":"2024-09-11T05:08:45.012434+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:45.012434+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":16631,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:42.411226+0000","last_change":"2024-09-11T05:08:32.586339+0000","last_active":"2024-09-11T05:08:42.411226+0000","last_peered":"2024-09-11T05:08:42.411226+0000","last_clean":"2024-09-11T04:55:08.774376+0000","last_became_active":"2024-09-11T04:56:20.617387+0000","last_became_peered":"2024-09-11T04:56:20.617387+0000","last_unstale":"2024-09-11T05:08:42.411226+0000","last_undegraded":"2024-09-11T04:56:20.287616+0000","last_fullsized":"2024-09-11T04:56:20.222916+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1905,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-414}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17702,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.919749+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:33.919749+0000","last_peered":"2024-09-11T05:08:33.919749+0000","last_clean":"2024-09-11T05:08:33.919749+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:33.919749+0000","last_undegraded":"2024-09-11T05:08:33.919749+0000","last_fullsized":"2024-09-11T05:08:33.919749+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12931,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.011851+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:45.011851+0000","last_peered":"2024-09-11T05:08:45.011851+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:45.011851+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17296,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-11T05:08:45.010558+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:45.010558+0000","last_peered":"2024-09-11T05:08:45.010558+0000","last_clean":"2024-09-11T05:08:45.010558+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:45.010558+0000","last_undegraded":"2024-09-11T05:08:45.010558+0000","last_fullsized":"2024-09-11T05:08:45.010558+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12791,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012679+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:45.012679+0000","last_peered":"2024-09-11T05:08:45.012679+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:45.012679+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.920029+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:33.920029+0000","last_peered":"2024-09-11T05:08:33.920029+0000","last_clean":"2024-09-11T05:08:33.920029+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:33.920029+0000","last_undegraded":"2024-09-11T05:08:33.920029+0000","last_fullsized":"2024-09-11T05:08:33.920029+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17520,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:08:33.918539+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:33.918539+0000","last_peered":"2024-09-11T05:08:33.918539+0000","last_clean":"2024-09-11T05:08:33.918539+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:33.918539+0000","last_undegraded":"2024-09-11T05:08:33.918539+0000","last_fullsized":"2024-09-11T05:08:33.918539+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":13076,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.399848+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:45.399848+0000","last_peered":"2024-09-11T05:08:45.399848+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:45.399848+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2087}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12550,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.011038+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:45.011038+0000","last_peered":"2024-09-11T05:08:45.011038+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:45.011038+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12917,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:33.920639+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:33.920639+0000","last_peered":"2024-09-11T05:08:33.920639+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:33.920639+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107776,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123829,"num_write_kb":453005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744272,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627568,"statfs":{"total":96636764160,"available":95874629632,"internally_reserved":0,"allocated":389120,"data_stored":147815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609592,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762248,"statfs":{"total":96636764160,"available":96012541952,"internally_reserved":0,"allocated":434176,"data_stored":197699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623767250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:47.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:47.952 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:47.953 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:47.967 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-11T05:08:47.940281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805756,"kb_used_data":1572,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681604,"statfs":{"total":386547056640,"available":383673962496,"internally_reserved":0,"allocated":1609728,"data_stored":639638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12726,"num_write_kb":-50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-696,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001386"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12632,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012902+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:45.012902+0000","last_peered":"2024-09-11T05:08:45.012902+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:45.012902+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12887,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:46.028708+0000","last_peered":"2024-09-11T05:08:46.028708+0000","last_clean":"2024-09-11T05:08:46.028708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:46.028708+0000","last_undegraded":"2024-09-11T05:08:46.028708+0000","last_fullsized":"2024-09-11T05:08:46.028708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12289,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.010274+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:45.010274+0000","last_peered":"2024-09-11T05:08:45.010274+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:45.010274+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13954,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028275+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:46.028275+0000","last_peered":"2024-09-11T05:08:46.028275+0000","last_clean":"2024-09-11T05:08:46.028275+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:46.028275+0000","last_undegraded":"2024-09-11T05:08:46.028275+0000","last_fullsized":"2024-09-11T05:08:46.028275+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17140,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-11T05:08:45.011234+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:45.011234+0000","last_peered":"2024-09-11T05:08:45.011234+0000","last_clean":"2024-09-11T05:08:45.011234+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:45.011234+0000","last_undegraded":"2024-09-11T05:08:45.011234+0000","last_fullsized":"2024-09-11T05:08:45.011234+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12769,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012434+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:45.012434+0000","last_peered":"2024-09-11T05:08:45.012434+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:45.012434+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17484,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.403995+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:46.403995+0000","last_peered":"2024-09-11T05:08:46.403995+0000","last_clean":"2024-09-11T05:08:46.403995+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:46.403995+0000","last_undegraded":"2024-09-11T05:08:46.403995+0000","last_fullsized":"2024-09-11T05:08:46.403995+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17706,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028889+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:46.028889+0000","last_peered":"2024-09-11T05:08:46.028889+0000","last_clean":"2024-09-11T05:08:46.028889+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:46.028889+0000","last_undegraded":"2024-09-11T05:08:46.028889+0000","last_fullsized":"2024-09-11T05:08:46.028889+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12931,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.011851+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:45.011851+0000","last_peered":"2024-09-11T05:08:45.011851+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:45.011851+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17296,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-11T05:08:45.010558+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:45.010558+0000","last_peered":"2024-09-11T05:08:45.010558+0000","last_clean":"2024-09-11T05:08:45.010558+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:45.010558+0000","last_undegraded":"2024-09-11T05:08:45.010558+0000","last_fullsized":"2024-09-11T05:08:45.010558+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12791,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012679+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:45.012679+0000","last_peered":"2024-09-11T05:08:45.012679+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:45.012679+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17011,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028760+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:46.028760+0000","last_peered":"2024-09-11T05:08:46.028760+0000","last_clean":"2024-09-11T05:08:46.028760+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:46.028760+0000","last_undegraded":"2024-09-11T05:08:46.028760+0000","last_fullsized":"2024-09-11T05:08:46.028760+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17524,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028409+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:46.028409+0000","last_peered":"2024-09-11T05:08:46.028409+0000","last_clean":"2024-09-11T05:08:46.028409+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:46.028409+0000","last_undegraded":"2024-09-11T05:08:46.028409+0000","last_fullsized":"2024-09-11T05:08:46.028409+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":13076,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.399848+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:45.399848+0000","last_peered":"2024-09-11T05:08:45.399848+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:45.399848+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2087}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12550,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.011038+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:45.011038+0000","last_peered":"2024-09-11T05:08:45.011038+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:45.011038+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12921,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.030833+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:46.030833+0000","last_peered":"2024-09-11T05:08:46.030833+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:46.030833+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744272,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627568,"statfs":{"total":96636764160,"available":95874629632,"internally_reserved":0,"allocated":389120,"data_stored":147815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617928,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753912,"statfs":{"total":96636764160,"available":96004005888,"internally_reserved":0,"allocated":442368,"data_stored":196214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:47.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:47.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12632, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:45.012902+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:45.012902+0000', 'last_peered': '2024-09-11T05:08:45.012902+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:45.012902+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:47.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:47.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12289, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:45.010274+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:45.010274+0000', 'last_peered': '2024-09-11T05:08:45.010274+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:45.010274+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:47.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:47.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12769, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:45.012434+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:45.012434+0000', 'last_peered': '2024-09-11T05:08:45.012434+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:45.012434+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:47.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12931, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:45.011851+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:45.011851+0000', 'last_peered': '2024-09-11T05:08:45.011851+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:45.011851+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12791, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:45.012679+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:45.012679+0000', 'last_peered': '2024-09-11T05:08:45.012679+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:45.012679+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 13076, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:45.399848+0000', 'last_change': '2024-09-11T05:08:44.289688+0000', 'last_active': '2024-09-11T05:08:45.399848+0000', 'last_peered': '2024-09-11T05:08:45.399848+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:45.399848+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2089, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 202, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12550, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:45.011038+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:45.011038+0000', 'last_peered': '2024-09-11T05:08:45.011038+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:45.011038+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:47.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12921, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.030833+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:46.030833+0000', 'last_peered': '2024-09-11T05:08:46.030833+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:46.030833+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:47.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18151/56 objects degraded (32412.500%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":855,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14,"data_bytes":94304,"bytes_used":2873094144,"bytes_avail":383673962496,"bytes_total":386547056640,"degraded_objects":18151,"degraded_total":56,"degraded_ratio":324.125},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:48.414 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:08:48.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:48.728 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:48.741 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-11T05:08:47.940281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805756,"kb_used_data":1572,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681604,"statfs":{"total":386547056640,"available":383673962496,"internally_reserved":0,"allocated":1609728,"data_stored":639638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12726,"num_write_kb":-50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-696,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001386"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12632,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012902+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:45.012902+0000","last_peered":"2024-09-11T05:08:45.012902+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:45.012902+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12887,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:46.028708+0000","last_peered":"2024-09-11T05:08:46.028708+0000","last_clean":"2024-09-11T05:08:46.028708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:46.028708+0000","last_undegraded":"2024-09-11T05:08:46.028708+0000","last_fullsized":"2024-09-11T05:08:46.028708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12289,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.010274+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:45.010274+0000","last_peered":"2024-09-11T05:08:45.010274+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:45.010274+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13954,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028275+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:46.028275+0000","last_peered":"2024-09-11T05:08:46.028275+0000","last_clean":"2024-09-11T05:08:46.028275+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:46.028275+0000","last_undegraded":"2024-09-11T05:08:46.028275+0000","last_fullsized":"2024-09-11T05:08:46.028275+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17140,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-11T05:08:45.011234+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:45.011234+0000","last_peered":"2024-09-11T05:08:45.011234+0000","last_clean":"2024-09-11T05:08:45.011234+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:45.011234+0000","last_undegraded":"2024-09-11T05:08:45.011234+0000","last_fullsized":"2024-09-11T05:08:45.011234+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12769,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012434+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:45.012434+0000","last_peered":"2024-09-11T05:08:45.012434+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:45.012434+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17484,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.403995+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:46.403995+0000","last_peered":"2024-09-11T05:08:46.403995+0000","last_clean":"2024-09-11T05:08:46.403995+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:46.403995+0000","last_undegraded":"2024-09-11T05:08:46.403995+0000","last_fullsized":"2024-09-11T05:08:46.403995+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17706,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028889+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:46.028889+0000","last_peered":"2024-09-11T05:08:46.028889+0000","last_clean":"2024-09-11T05:08:46.028889+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:46.028889+0000","last_undegraded":"2024-09-11T05:08:46.028889+0000","last_fullsized":"2024-09-11T05:08:46.028889+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12931,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.011851+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:45.011851+0000","last_peered":"2024-09-11T05:08:45.011851+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:45.011851+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17296,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-11T05:08:45.010558+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:45.010558+0000","last_peered":"2024-09-11T05:08:45.010558+0000","last_clean":"2024-09-11T05:08:45.010558+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:45.010558+0000","last_undegraded":"2024-09-11T05:08:45.010558+0000","last_fullsized":"2024-09-11T05:08:45.010558+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12791,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.012679+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:45.012679+0000","last_peered":"2024-09-11T05:08:45.012679+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:45.012679+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17011,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028760+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:46.028760+0000","last_peered":"2024-09-11T05:08:46.028760+0000","last_clean":"2024-09-11T05:08:46.028760+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:46.028760+0000","last_undegraded":"2024-09-11T05:08:46.028760+0000","last_fullsized":"2024-09-11T05:08:46.028760+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17524,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028409+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:46.028409+0000","last_peered":"2024-09-11T05:08:46.028409+0000","last_clean":"2024-09-11T05:08:46.028409+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:46.028409+0000","last_undegraded":"2024-09-11T05:08:46.028409+0000","last_fullsized":"2024-09-11T05:08:46.028409+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":13076,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.399848+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:45.399848+0000","last_peered":"2024-09-11T05:08:45.399848+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:45.399848+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2087}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12550,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:45.011038+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:45.011038+0000","last_peered":"2024-09-11T05:08:45.011038+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:45.011038+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12921,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.030833+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:46.030833+0000","last_peered":"2024-09-11T05:08:46.030833+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:46.030833+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744272,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627568,"statfs":{"total":96636764160,"available":95874629632,"internally_reserved":0,"allocated":389120,"data_stored":147815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721544,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650296,"statfs":{"total":96636764160,"available":95897903104,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617928,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753912,"statfs":{"total":96636764160,"available":96004005888,"internally_reserved":0,"allocated":442368,"data_stored":196214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:51.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:52.082 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:52.082 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:52.097 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-11T05:08:51.940943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805780,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681580,"statfs":{"total":386547056640,"available":383673937920,"internally_reserved":0,"allocated":1634304,"data_stored":649688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12726,"num_write_kb":-50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-907,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001378"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12634,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032876+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:46.032876+0000","last_peered":"2024-09-11T05:08:46.032876+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:46.032876+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12887,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:46.028708+0000","last_peered":"2024-09-11T05:08:46.028708+0000","last_clean":"2024-09-11T05:08:46.028708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:46.028708+0000","last_undegraded":"2024-09-11T05:08:46.028708+0000","last_fullsized":"2024-09-11T05:08:46.028708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12291,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029317+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:46.029317+0000","last_peered":"2024-09-11T05:08:46.029317+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:46.029317+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13954,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028275+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:46.028275+0000","last_peered":"2024-09-11T05:08:46.028275+0000","last_clean":"2024-09-11T05:08:46.028275+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:46.028275+0000","last_undegraded":"2024-09-11T05:08:46.028275+0000","last_fullsized":"2024-09-11T05:08:46.028275+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17142,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.032631+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:46.032631+0000","last_peered":"2024-09-11T05:08:46.032631+0000","last_clean":"2024-09-11T05:08:46.032631+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:46.032631+0000","last_undegraded":"2024-09-11T05:08:46.032631+0000","last_fullsized":"2024-09-11T05:08:46.032631+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12771,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032865+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:46.032865+0000","last_peered":"2024-09-11T05:08:46.032865+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:46.032865+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17484,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.403995+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:46.403995+0000","last_peered":"2024-09-11T05:08:46.403995+0000","last_clean":"2024-09-11T05:08:46.403995+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:46.403995+0000","last_undegraded":"2024-09-11T05:08:46.403995+0000","last_fullsized":"2024-09-11T05:08:46.403995+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17706,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028889+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:46.028889+0000","last_peered":"2024-09-11T05:08:46.028889+0000","last_clean":"2024-09-11T05:08:46.028889+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:46.028889+0000","last_undegraded":"2024-09-11T05:08:46.028889+0000","last_fullsized":"2024-09-11T05:08:46.028889+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12933,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032853+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:46.032853+0000","last_peered":"2024-09-11T05:08:46.032853+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:46.032853+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17298,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.029816+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:46.029816+0000","last_peered":"2024-09-11T05:08:46.029816+0000","last_clean":"2024-09-11T05:08:46.029816+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:46.029816+0000","last_undegraded":"2024-09-11T05:08:46.029816+0000","last_fullsized":"2024-09-11T05:08:46.029816+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032746+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:46.032746+0000","last_peered":"2024-09-11T05:08:46.032746+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:46.032746+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17011,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028760+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:46.028760+0000","last_peered":"2024-09-11T05:08:46.028760+0000","last_clean":"2024-09-11T05:08:46.028760+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:46.028760+0000","last_undegraded":"2024-09-11T05:08:46.028760+0000","last_fullsized":"2024-09-11T05:08:46.028760+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17524,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028409+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:46.028409+0000","last_peered":"2024-09-11T05:08:46.028409+0000","last_clean":"2024-09-11T05:08:46.028409+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:46.028409+0000","last_undegraded":"2024-09-11T05:08:46.028409+0000","last_fullsized":"2024-09-11T05:08:46.028409+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":14720,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:50.400010+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:50.400010+0000","last_peered":"2024-09-11T05:08:50.400010+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:50.400010+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1266}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12552,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029135+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:46.029135+0000","last_peered":"2024-09-11T05:08:46.029135+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:46.029135+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12921,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.030833+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:46.030833+0000","last_peered":"2024-09-11T05:08:46.030833+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:46.030833+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744280,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627560,"statfs":{"total":96636764160,"available":95874621440,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721560,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650280,"statfs":{"total":96636764160,"available":95897886720,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617928,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753912,"statfs":{"total":96636764160,"available":96004005888,"internally_reserved":0,"allocated":442368,"data_stored":196214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:52.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:52.417 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:52.432 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-11T05:08:51.940943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805780,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681580,"statfs":{"total":386547056640,"available":383673937920,"internally_reserved":0,"allocated":1634304,"data_stored":649688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12726,"num_write_kb":-50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-907,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001378"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12634,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032876+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:46.032876+0000","last_peered":"2024-09-11T05:08:46.032876+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:46.032876+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12887,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:46.028708+0000","last_peered":"2024-09-11T05:08:46.028708+0000","last_clean":"2024-09-11T05:08:46.028708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:46.028708+0000","last_undegraded":"2024-09-11T05:08:46.028708+0000","last_fullsized":"2024-09-11T05:08:46.028708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12291,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029317+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:46.029317+0000","last_peered":"2024-09-11T05:08:46.029317+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:46.029317+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13954,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028275+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:46.028275+0000","last_peered":"2024-09-11T05:08:46.028275+0000","last_clean":"2024-09-11T05:08:46.028275+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:46.028275+0000","last_undegraded":"2024-09-11T05:08:46.028275+0000","last_fullsized":"2024-09-11T05:08:46.028275+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17142,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.032631+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:46.032631+0000","last_peered":"2024-09-11T05:08:46.032631+0000","last_clean":"2024-09-11T05:08:46.032631+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:46.032631+0000","last_undegraded":"2024-09-11T05:08:46.032631+0000","last_fullsized":"2024-09-11T05:08:46.032631+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12771,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032865+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:46.032865+0000","last_peered":"2024-09-11T05:08:46.032865+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:46.032865+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17484,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.403995+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:46.403995+0000","last_peered":"2024-09-11T05:08:46.403995+0000","last_clean":"2024-09-11T05:08:46.403995+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:46.403995+0000","last_undegraded":"2024-09-11T05:08:46.403995+0000","last_fullsized":"2024-09-11T05:08:46.403995+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17706,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028889+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:46.028889+0000","last_peered":"2024-09-11T05:08:46.028889+0000","last_clean":"2024-09-11T05:08:46.028889+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:46.028889+0000","last_undegraded":"2024-09-11T05:08:46.028889+0000","last_fullsized":"2024-09-11T05:08:46.028889+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12933,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032853+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:46.032853+0000","last_peered":"2024-09-11T05:08:46.032853+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:46.032853+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17298,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.029816+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:46.029816+0000","last_peered":"2024-09-11T05:08:46.029816+0000","last_clean":"2024-09-11T05:08:46.029816+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:46.029816+0000","last_undegraded":"2024-09-11T05:08:46.029816+0000","last_fullsized":"2024-09-11T05:08:46.029816+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032746+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:46.032746+0000","last_peered":"2024-09-11T05:08:46.032746+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:46.032746+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17011,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028760+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:46.028760+0000","last_peered":"2024-09-11T05:08:46.028760+0000","last_clean":"2024-09-11T05:08:46.028760+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:46.028760+0000","last_undegraded":"2024-09-11T05:08:46.028760+0000","last_fullsized":"2024-09-11T05:08:46.028760+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17524,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028409+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:46.028409+0000","last_peered":"2024-09-11T05:08:46.028409+0000","last_clean":"2024-09-11T05:08:46.028409+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:46.028409+0000","last_undegraded":"2024-09-11T05:08:46.028409+0000","last_fullsized":"2024-09-11T05:08:46.028409+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":14720,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:50.400010+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:50.400010+0000","last_peered":"2024-09-11T05:08:50.400010+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:50.400010+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1266}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12552,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029135+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:46.029135+0000","last_peered":"2024-09-11T05:08:46.029135+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:46.029135+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12921,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.030833+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:46.030833+0000","last_peered":"2024-09-11T05:08:46.030833+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:46.030833+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744280,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627560,"statfs":{"total":96636764160,"available":95874621440,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721560,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650280,"statfs":{"total":96636764160,"available":95897886720,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617928,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753912,"statfs":{"total":96636764160,"available":96004005888,"internally_reserved":0,"allocated":442368,"data_stored":196214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:52.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:52.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12634, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032876+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:46.032876+0000', 'last_peered': '2024-09-11T05:08:46.032876+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:46.032876+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:52.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:52.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12291, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.029317+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:46.029317+0000', 'last_peered': '2024-09-11T05:08:46.029317+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:46.029317+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:52.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12771, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032865+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:46.032865+0000', 'last_peered': '2024-09-11T05:08:46.032865+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:46.032865+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12933, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032853+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:46.032853+0000', 'last_peered': '2024-09-11T05:08:46.032853+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:46.032853+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12793, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032746+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:46.032746+0000', 'last_peered': '2024-09-11T05:08:46.032746+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:46.032746+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 14720, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:50.400010+0000', 'last_change': '2024-09-11T05:08:44.289688+0000', 'last_active': '2024-09-11T05:08:50.400010+0000', 'last_peered': '2024-09-11T05:08:50.400010+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:50.400010+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1023, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12552, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.029135+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:46.029135+0000', 'last_peered': '2024-09-11T05:08:46.029135+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:46.029135+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:52.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12921, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.030833+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:46.030833+0000', 'last_peered': '2024-09-11T05:08:46.030833+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:46.030833+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:52.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:52.869 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:52.870 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17330/56 objects degraded (30946.429%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":859,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14,"data_bytes":94304,"bytes_used":2873118720,"bytes_avail":383673937920,"bytes_total":386547056640,"degraded_objects":17330,"degraded_total":56,"degraded_ratio":309.46428571428572},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:52.884 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:08:52.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:53.223 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:53.223 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:53.236 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-11T05:08:51.940943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805780,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681580,"statfs":{"total":386547056640,"available":383673937920,"internally_reserved":0,"allocated":1634304,"data_stored":649688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12726,"num_write_kb":-50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-907,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001378"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12634,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032876+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:46.032876+0000","last_peered":"2024-09-11T05:08:46.032876+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:46.032876+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12887,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:46.028708+0000","last_peered":"2024-09-11T05:08:46.028708+0000","last_clean":"2024-09-11T05:08:46.028708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:46.028708+0000","last_undegraded":"2024-09-11T05:08:46.028708+0000","last_fullsized":"2024-09-11T05:08:46.028708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12291,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029317+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:46.029317+0000","last_peered":"2024-09-11T05:08:46.029317+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:46.029317+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13954,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028275+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:46.028275+0000","last_peered":"2024-09-11T05:08:46.028275+0000","last_clean":"2024-09-11T05:08:46.028275+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:46.028275+0000","last_undegraded":"2024-09-11T05:08:46.028275+0000","last_fullsized":"2024-09-11T05:08:46.028275+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17142,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.032631+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:46.032631+0000","last_peered":"2024-09-11T05:08:46.032631+0000","last_clean":"2024-09-11T05:08:46.032631+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:46.032631+0000","last_undegraded":"2024-09-11T05:08:46.032631+0000","last_fullsized":"2024-09-11T05:08:46.032631+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12771,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032865+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:46.032865+0000","last_peered":"2024-09-11T05:08:46.032865+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:46.032865+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17484,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.403995+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:46.403995+0000","last_peered":"2024-09-11T05:08:46.403995+0000","last_clean":"2024-09-11T05:08:46.403995+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:46.403995+0000","last_undegraded":"2024-09-11T05:08:46.403995+0000","last_fullsized":"2024-09-11T05:08:46.403995+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17706,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028889+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:46.028889+0000","last_peered":"2024-09-11T05:08:46.028889+0000","last_clean":"2024-09-11T05:08:46.028889+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:46.028889+0000","last_undegraded":"2024-09-11T05:08:46.028889+0000","last_fullsized":"2024-09-11T05:08:46.028889+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12933,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032853+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:46.032853+0000","last_peered":"2024-09-11T05:08:46.032853+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:46.032853+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17298,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.029816+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:46.029816+0000","last_peered":"2024-09-11T05:08:46.029816+0000","last_clean":"2024-09-11T05:08:46.029816+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:46.029816+0000","last_undegraded":"2024-09-11T05:08:46.029816+0000","last_fullsized":"2024-09-11T05:08:46.029816+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032746+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:46.032746+0000","last_peered":"2024-09-11T05:08:46.032746+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:46.032746+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17011,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028760+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:46.028760+0000","last_peered":"2024-09-11T05:08:46.028760+0000","last_clean":"2024-09-11T05:08:46.028760+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:46.028760+0000","last_undegraded":"2024-09-11T05:08:46.028760+0000","last_fullsized":"2024-09-11T05:08:46.028760+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17524,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028409+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:46.028409+0000","last_peered":"2024-09-11T05:08:46.028409+0000","last_clean":"2024-09-11T05:08:46.028409+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:46.028409+0000","last_undegraded":"2024-09-11T05:08:46.028409+0000","last_fullsized":"2024-09-11T05:08:46.028409+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":14720,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:50.400010+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:50.400010+0000","last_peered":"2024-09-11T05:08:50.400010+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:50.400010+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1266}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12552,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029135+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:46.029135+0000","last_peered":"2024-09-11T05:08:46.029135+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:46.029135+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12921,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.030833+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:46.030833+0000","last_peered":"2024-09-11T05:08:46.030833+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:46.030833+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744280,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627560,"statfs":{"total":96636764160,"available":95874621440,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721560,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650280,"statfs":{"total":96636764160,"available":95897886720,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617928,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753912,"statfs":{"total":96636764160,"available":96004005888,"internally_reserved":0,"allocated":442368,"data_stored":196214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:56.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:56.580 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:56.581 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:56.593 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-11T05:08:55.941638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797444,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2795761,"kb_avail":374689916,"statfs":{"total":386547056640,"available":383682473984,"internally_reserved":0,"allocated":1634304,"data_stored":649688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2862859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12726,"num_write_kb":-50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001719"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12634,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032876+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:46.032876+0000","last_peered":"2024-09-11T05:08:46.032876+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:46.032876+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12887,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:46.028708+0000","last_peered":"2024-09-11T05:08:46.028708+0000","last_clean":"2024-09-11T05:08:46.028708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:46.028708+0000","last_undegraded":"2024-09-11T05:08:46.028708+0000","last_fullsized":"2024-09-11T05:08:46.028708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12291,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029317+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:46.029317+0000","last_peered":"2024-09-11T05:08:46.029317+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:46.029317+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13954,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028275+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:46.028275+0000","last_peered":"2024-09-11T05:08:46.028275+0000","last_clean":"2024-09-11T05:08:46.028275+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:46.028275+0000","last_undegraded":"2024-09-11T05:08:46.028275+0000","last_fullsized":"2024-09-11T05:08:46.028275+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17142,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.032631+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:46.032631+0000","last_peered":"2024-09-11T05:08:46.032631+0000","last_clean":"2024-09-11T05:08:46.032631+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:46.032631+0000","last_undegraded":"2024-09-11T05:08:46.032631+0000","last_fullsized":"2024-09-11T05:08:46.032631+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12771,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032865+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:46.032865+0000","last_peered":"2024-09-11T05:08:46.032865+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:46.032865+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17484,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.403995+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:46.403995+0000","last_peered":"2024-09-11T05:08:46.403995+0000","last_clean":"2024-09-11T05:08:46.403995+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:46.403995+0000","last_undegraded":"2024-09-11T05:08:46.403995+0000","last_fullsized":"2024-09-11T05:08:46.403995+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17706,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028889+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:46.028889+0000","last_peered":"2024-09-11T05:08:46.028889+0000","last_clean":"2024-09-11T05:08:46.028889+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:46.028889+0000","last_undegraded":"2024-09-11T05:08:46.028889+0000","last_fullsized":"2024-09-11T05:08:46.028889+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12933,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032853+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:46.032853+0000","last_peered":"2024-09-11T05:08:46.032853+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:46.032853+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17298,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.029816+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:46.029816+0000","last_peered":"2024-09-11T05:08:46.029816+0000","last_clean":"2024-09-11T05:08:46.029816+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:46.029816+0000","last_undegraded":"2024-09-11T05:08:46.029816+0000","last_fullsized":"2024-09-11T05:08:46.029816+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032746+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:46.032746+0000","last_peered":"2024-09-11T05:08:46.032746+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:46.032746+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17011,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028760+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:46.028760+0000","last_peered":"2024-09-11T05:08:46.028760+0000","last_clean":"2024-09-11T05:08:46.028760+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:46.028760+0000","last_undegraded":"2024-09-11T05:08:46.028760+0000","last_fullsized":"2024-09-11T05:08:46.028760+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17525,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028457+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:46.028457+0000","last_peered":"2024-09-11T05:08:46.028457+0000","last_clean":"2024-09-11T05:08:46.028457+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:46.028457+0000","last_undegraded":"2024-09-11T05:08:46.028457+0000","last_fullsized":"2024-09-11T05:08:46.028457+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":16742,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:55.402797+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:55.402797+0000","last_peered":"2024-09-11T05:08:55.402797+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:55.402797+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12552,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029135+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:46.029135+0000","last_peered":"2024-09-11T05:08:46.029135+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:46.029135+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12921,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.030833+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:46.030833+0000","last_peered":"2024-09-11T05:08:46.030833+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:46.030833+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744280,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627560,"statfs":{"total":96636764160,"available":95874621440,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721560,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650280,"statfs":{"total":96636764160,"available":95897886720,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609600,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762240,"statfs":{"total":96636764160,"available":96012533760,"internally_reserved":0,"allocated":442368,"data_stored":196214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623767250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:56.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:56.910 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:56.910 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:56.923 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-11T05:08:55.941638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797444,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2795761,"kb_avail":374689916,"statfs":{"total":386547056640,"available":383682473984,"internally_reserved":0,"allocated":1634304,"data_stored":649688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2862859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12726,"num_write_kb":-50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001719"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12634,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032876+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:46.032876+0000","last_peered":"2024-09-11T05:08:46.032876+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:46.032876+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12887,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:46.028708+0000","last_peered":"2024-09-11T05:08:46.028708+0000","last_clean":"2024-09-11T05:08:46.028708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:46.028708+0000","last_undegraded":"2024-09-11T05:08:46.028708+0000","last_fullsized":"2024-09-11T05:08:46.028708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12291,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029317+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:46.029317+0000","last_peered":"2024-09-11T05:08:46.029317+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:46.029317+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13954,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028275+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:46.028275+0000","last_peered":"2024-09-11T05:08:46.028275+0000","last_clean":"2024-09-11T05:08:46.028275+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:46.028275+0000","last_undegraded":"2024-09-11T05:08:46.028275+0000","last_fullsized":"2024-09-11T05:08:46.028275+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17142,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.032631+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:46.032631+0000","last_peered":"2024-09-11T05:08:46.032631+0000","last_clean":"2024-09-11T05:08:46.032631+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:46.032631+0000","last_undegraded":"2024-09-11T05:08:46.032631+0000","last_fullsized":"2024-09-11T05:08:46.032631+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12771,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032865+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:46.032865+0000","last_peered":"2024-09-11T05:08:46.032865+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:46.032865+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17484,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.403995+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:46.403995+0000","last_peered":"2024-09-11T05:08:46.403995+0000","last_clean":"2024-09-11T05:08:46.403995+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:46.403995+0000","last_undegraded":"2024-09-11T05:08:46.403995+0000","last_fullsized":"2024-09-11T05:08:46.403995+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17706,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028889+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:46.028889+0000","last_peered":"2024-09-11T05:08:46.028889+0000","last_clean":"2024-09-11T05:08:46.028889+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:46.028889+0000","last_undegraded":"2024-09-11T05:08:46.028889+0000","last_fullsized":"2024-09-11T05:08:46.028889+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12933,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032853+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:46.032853+0000","last_peered":"2024-09-11T05:08:46.032853+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:46.032853+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17298,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.029816+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:46.029816+0000","last_peered":"2024-09-11T05:08:46.029816+0000","last_clean":"2024-09-11T05:08:46.029816+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:46.029816+0000","last_undegraded":"2024-09-11T05:08:46.029816+0000","last_fullsized":"2024-09-11T05:08:46.029816+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032746+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:46.032746+0000","last_peered":"2024-09-11T05:08:46.032746+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:46.032746+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17011,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028760+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:46.028760+0000","last_peered":"2024-09-11T05:08:46.028760+0000","last_clean":"2024-09-11T05:08:46.028760+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:46.028760+0000","last_undegraded":"2024-09-11T05:08:46.028760+0000","last_fullsized":"2024-09-11T05:08:46.028760+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17525,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028457+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:46.028457+0000","last_peered":"2024-09-11T05:08:46.028457+0000","last_clean":"2024-09-11T05:08:46.028457+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:46.028457+0000","last_undegraded":"2024-09-11T05:08:46.028457+0000","last_fullsized":"2024-09-11T05:08:46.028457+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":16742,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:55.402797+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:55.402797+0000","last_peered":"2024-09-11T05:08:55.402797+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:55.402797+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12552,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029135+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:46.029135+0000","last_peered":"2024-09-11T05:08:46.029135+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:46.029135+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12921,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.030833+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:46.030833+0000","last_peered":"2024-09-11T05:08:46.030833+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:46.030833+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744280,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627560,"statfs":{"total":96636764160,"available":95874621440,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721560,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650280,"statfs":{"total":96636764160,"available":95897886720,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609600,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762240,"statfs":{"total":96636764160,"available":96012533760,"internally_reserved":0,"allocated":442368,"data_stored":196214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623767250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:08:56.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:08:56.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12634, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032876+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:46.032876+0000', 'last_peered': '2024-09-11T05:08:46.032876+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:46.032876+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:56.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:08:56.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12291, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.029317+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:46.029317+0000', 'last_peered': '2024-09-11T05:08:46.029317+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:46.029317+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:56.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:08:56.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12771, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032865+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:46.032865+0000', 'last_peered': '2024-09-11T05:08:46.032865+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:46.032865+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:56.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12933, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032853+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:46.032853+0000', 'last_peered': '2024-09-11T05:08:46.032853+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:46.032853+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12793, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032746+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:46.032746+0000', 'last_peered': '2024-09-11T05:08:46.032746+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:46.032746+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 16742, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:55.402797+0000', 'last_change': '2024-09-11T05:08:44.289688+0000', 'last_active': '2024-09-11T05:08:55.402797+0000', 'last_peered': '2024-09-11T05:08:55.402797+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:08:55.402797+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2034, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12552, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.029135+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:46.029135+0000', 'last_peered': '2024-09-11T05:08:46.029135+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:46.029135+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:08:56.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 12921, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.030833+0000', 'last_change': '2024-09-11T04:56:20.655601+0000', 'last_active': '2024-09-11T05:08:46.030833+0000', 'last_peered': '2024-09-11T05:08:46.030833+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:46.030833+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:08:56.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:08:57.350 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:57.350 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16319/56 objects degraded (29141.071%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":864,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14,"data_bytes":94304,"bytes_used":2864582656,"bytes_avail":383682473984,"bytes_total":386547056640,"degraded_objects":16319,"degraded_total":56,"degraded_ratio":291.41071428571428},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:08:57.364 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:08:57.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:08:57.734 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:08:57.735 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:08:57.747 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-11T05:08:57.064749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797444,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2795761,"kb_avail":374689916,"statfs":{"total":386547056640,"available":383682473984,"internally_reserved":0,"allocated":1634304,"data_stored":649688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2862859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12726,"num_write_kb":-50908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.037470"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12634,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032876+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:46.032876+0000","last_peered":"2024-09-11T05:08:46.032876+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:46.032876+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12887,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028708+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:46.028708+0000","last_peered":"2024-09-11T05:08:46.028708+0000","last_clean":"2024-09-11T05:08:46.028708+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:46.028708+0000","last_undegraded":"2024-09-11T05:08:46.028708+0000","last_fullsized":"2024-09-11T05:08:46.028708+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12291,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029317+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:46.029317+0000","last_peered":"2024-09-11T05:08:46.029317+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:46.029317+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13954,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028275+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:46.028275+0000","last_peered":"2024-09-11T05:08:46.028275+0000","last_clean":"2024-09-11T05:08:46.028275+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:46.028275+0000","last_undegraded":"2024-09-11T05:08:46.028275+0000","last_fullsized":"2024-09-11T05:08:46.028275+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17142,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.032631+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:46.032631+0000","last_peered":"2024-09-11T05:08:46.032631+0000","last_clean":"2024-09-11T05:08:46.032631+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:46.032631+0000","last_undegraded":"2024-09-11T05:08:46.032631+0000","last_fullsized":"2024-09-11T05:08:46.032631+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12771,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032865+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:46.032865+0000","last_peered":"2024-09-11T05:08:46.032865+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:46.032865+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17484,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.403995+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:46.403995+0000","last_peered":"2024-09-11T05:08:46.403995+0000","last_clean":"2024-09-11T05:08:46.403995+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:46.403995+0000","last_undegraded":"2024-09-11T05:08:46.403995+0000","last_fullsized":"2024-09-11T05:08:46.403995+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17706,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028889+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:46.028889+0000","last_peered":"2024-09-11T05:08:46.028889+0000","last_clean":"2024-09-11T05:08:46.028889+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:46.028889+0000","last_undegraded":"2024-09-11T05:08:46.028889+0000","last_fullsized":"2024-09-11T05:08:46.028889+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12933,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032853+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:46.032853+0000","last_peered":"2024-09-11T05:08:46.032853+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:46.032853+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17298,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.029816+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:46.029816+0000","last_peered":"2024-09-11T05:08:46.029816+0000","last_clean":"2024-09-11T05:08:46.029816+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:46.029816+0000","last_undegraded":"2024-09-11T05:08:46.029816+0000","last_fullsized":"2024-09-11T05:08:46.029816+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032746+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:46.032746+0000","last_peered":"2024-09-11T05:08:46.032746+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:46.032746+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17011,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028760+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:46.028760+0000","last_peered":"2024-09-11T05:08:46.028760+0000","last_clean":"2024-09-11T05:08:46.028760+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:46.028760+0000","last_undegraded":"2024-09-11T05:08:46.028760+0000","last_fullsized":"2024-09-11T05:08:46.028760+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17525,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.028457+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:46.028457+0000","last_peered":"2024-09-11T05:08:46.028457+0000","last_clean":"2024-09-11T05:08:46.028457+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:46.028457+0000","last_undegraded":"2024-09-11T05:08:46.028457+0000","last_fullsized":"2024-09-11T05:08:46.028457+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":16742,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:55.402797+0000","last_change":"2024-09-11T05:08:44.289688+0000","last_active":"2024-09-11T05:08:55.402797+0000","last_peered":"2024-09-11T05:08:55.402797+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:08:55.402797+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12552,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029135+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:46.029135+0000","last_peered":"2024-09-11T05:08:46.029135+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:46.029135+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12921,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.030833+0000","last_change":"2024-09-11T04:56:20.655601+0000","last_active":"2024-09-11T05:08:46.030833+0000","last_peered":"2024-09-11T05:08:46.030833+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:46.030833+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744280,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627560,"statfs":{"total":96636764160,"available":95874621440,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721560,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650280,"statfs":{"total":96636764160,"available":95897886720,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609600,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762240,"statfs":{"total":96636764160,"available":96012533760,"internally_reserved":0,"allocated":442368,"data_stored":196214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":623767250},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:00.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:01.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:01.076 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:01.088 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-11T05:08:59.942359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805756,"kb_used_data":1588,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681604,"statfs":{"total":386547056640,"available":383673962496,"internally_reserved":0,"allocated":1626112,"data_stored":641496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-848,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001415"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12634,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032876+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:46.032876+0000","last_peered":"2024-09-11T05:08:46.032876+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:46.032876+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12889,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070801+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:57.070801+0000","last_peered":"2024-09-11T05:08:57.070801+0000","last_clean":"2024-09-11T05:08:57.070801+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:57.070801+0000","last_undegraded":"2024-09-11T05:08:57.070801+0000","last_fullsized":"2024-09-11T05:08:57.070801+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12291,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029317+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:46.029317+0000","last_peered":"2024-09-11T05:08:46.029317+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:46.029317+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13956,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070608+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:57.070608+0000","last_peered":"2024-09-11T05:08:57.070608+0000","last_clean":"2024-09-11T05:08:57.070608+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:57.070608+0000","last_undegraded":"2024-09-11T05:08:57.070608+0000","last_fullsized":"2024-09-11T05:08:57.070608+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17142,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.032631+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:46.032631+0000","last_peered":"2024-09-11T05:08:46.032631+0000","last_clean":"2024-09-11T05:08:46.032631+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:46.032631+0000","last_undegraded":"2024-09-11T05:08:46.032631+0000","last_fullsized":"2024-09-11T05:08:46.032631+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12771,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032865+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:46.032865+0000","last_peered":"2024-09-11T05:08:46.032865+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:46.032865+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17486,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.067825+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:57.067825+0000","last_peered":"2024-09-11T05:08:57.067825+0000","last_clean":"2024-09-11T05:08:57.067825+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:57.067825+0000","last_undegraded":"2024-09-11T05:08:57.067825+0000","last_fullsized":"2024-09-11T05:08:57.067825+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17708,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.071464+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:57.071464+0000","last_peered":"2024-09-11T05:08:57.071464+0000","last_clean":"2024-09-11T05:08:57.071464+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:57.071464+0000","last_undegraded":"2024-09-11T05:08:57.071464+0000","last_fullsized":"2024-09-11T05:08:57.071464+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12933,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032853+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:46.032853+0000","last_peered":"2024-09-11T05:08:46.032853+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:46.032853+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17298,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.029816+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:46.029816+0000","last_peered":"2024-09-11T05:08:46.029816+0000","last_clean":"2024-09-11T05:08:46.029816+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:46.029816+0000","last_undegraded":"2024-09-11T05:08:46.029816+0000","last_fullsized":"2024-09-11T05:08:46.029816+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032746+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:46.032746+0000","last_peered":"2024-09-11T05:08:46.032746+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:46.032746+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17013,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070909+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:57.070909+0000","last_peered":"2024-09-11T05:08:57.070909+0000","last_clean":"2024-09-11T05:08:57.070909+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:57.070909+0000","last_undegraded":"2024-09-11T05:08:57.070909+0000","last_fullsized":"2024-09-11T05:08:57.070909+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17527,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070636+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:57.070636+0000","last_peered":"2024-09-11T05:08:57.070636+0000","last_clean":"2024-09-11T05:08:57.070636+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:57.070636+0000","last_undegraded":"2024-09-11T05:08:57.070636+0000","last_fullsized":"2024-09-11T05:08:57.070636+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-11T05:07:50.442739+0000","last_change":"2024-09-11T05:08:57.093579+0000","last_active":"2024-09-11T05:08:57.093579+0000","last_peered":"2024-09-11T05:07:50.442739+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.442739+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12552,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029135+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:46.029135+0000","last_peered":"2024-09-11T05:08:46.029135+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:46.029135+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":13224,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:57.410453+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:08:57.410453+0000","last_peered":"2024-09-11T05:08:57.410453+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:57.410453+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744280,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627560,"statfs":{"total":96636764160,"available":95874621440,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721560,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650280,"statfs":{"total":96636764160,"available":95897886720,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617912,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753928,"statfs":{"total":96636764160,"available":96004022272,"internally_reserved":0,"allocated":434176,"data_stored":188022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:01.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:01.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:01.405 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:01.417 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-11T05:08:59.942359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805756,"kb_used_data":1588,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681604,"statfs":{"total":386547056640,"available":383673962496,"internally_reserved":0,"allocated":1626112,"data_stored":641496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-848,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001415"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12634,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032876+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:46.032876+0000","last_peered":"2024-09-11T05:08:46.032876+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:46.032876+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12889,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070801+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:57.070801+0000","last_peered":"2024-09-11T05:08:57.070801+0000","last_clean":"2024-09-11T05:08:57.070801+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:57.070801+0000","last_undegraded":"2024-09-11T05:08:57.070801+0000","last_fullsized":"2024-09-11T05:08:57.070801+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12291,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029317+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:46.029317+0000","last_peered":"2024-09-11T05:08:46.029317+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:46.029317+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13956,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070608+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:57.070608+0000","last_peered":"2024-09-11T05:08:57.070608+0000","last_clean":"2024-09-11T05:08:57.070608+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:57.070608+0000","last_undegraded":"2024-09-11T05:08:57.070608+0000","last_fullsized":"2024-09-11T05:08:57.070608+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17142,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.032631+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:46.032631+0000","last_peered":"2024-09-11T05:08:46.032631+0000","last_clean":"2024-09-11T05:08:46.032631+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:46.032631+0000","last_undegraded":"2024-09-11T05:08:46.032631+0000","last_fullsized":"2024-09-11T05:08:46.032631+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12771,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032865+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:46.032865+0000","last_peered":"2024-09-11T05:08:46.032865+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:46.032865+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17486,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.067825+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:57.067825+0000","last_peered":"2024-09-11T05:08:57.067825+0000","last_clean":"2024-09-11T05:08:57.067825+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:57.067825+0000","last_undegraded":"2024-09-11T05:08:57.067825+0000","last_fullsized":"2024-09-11T05:08:57.067825+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17708,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.071464+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:57.071464+0000","last_peered":"2024-09-11T05:08:57.071464+0000","last_clean":"2024-09-11T05:08:57.071464+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:57.071464+0000","last_undegraded":"2024-09-11T05:08:57.071464+0000","last_fullsized":"2024-09-11T05:08:57.071464+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12933,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032853+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:46.032853+0000","last_peered":"2024-09-11T05:08:46.032853+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:46.032853+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17298,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-11T05:08:46.029816+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:46.029816+0000","last_peered":"2024-09-11T05:08:46.029816+0000","last_clean":"2024-09-11T05:08:46.029816+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:46.029816+0000","last_undegraded":"2024-09-11T05:08:46.029816+0000","last_fullsized":"2024-09-11T05:08:46.029816+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.032746+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:46.032746+0000","last_peered":"2024-09-11T05:08:46.032746+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:46.032746+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17013,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070909+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:57.070909+0000","last_peered":"2024-09-11T05:08:57.070909+0000","last_clean":"2024-09-11T05:08:57.070909+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:57.070909+0000","last_undegraded":"2024-09-11T05:08:57.070909+0000","last_fullsized":"2024-09-11T05:08:57.070909+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17527,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070636+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:57.070636+0000","last_peered":"2024-09-11T05:08:57.070636+0000","last_clean":"2024-09-11T05:08:57.070636+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:57.070636+0000","last_undegraded":"2024-09-11T05:08:57.070636+0000","last_fullsized":"2024-09-11T05:08:57.070636+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-11T05:07:50.442739+0000","last_change":"2024-09-11T05:08:57.093579+0000","last_active":"2024-09-11T05:08:57.093579+0000","last_peered":"2024-09-11T05:07:50.442739+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.442739+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12552,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:46.029135+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:46.029135+0000","last_peered":"2024-09-11T05:08:46.029135+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:46.029135+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":13224,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:57.410453+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:08:57.410453+0000","last_peered":"2024-09-11T05:08:57.410453+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:57.410453+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744280,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627560,"statfs":{"total":96636764160,"available":95874621440,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721560,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650280,"statfs":{"total":96636764160,"available":95897886720,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617912,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753928,"statfs":{"total":96636764160,"available":96004022272,"internally_reserved":0,"allocated":434176,"data_stored":188022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:01.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:01.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12634, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032876+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:46.032876+0000', 'last_peered': '2024-09-11T05:08:46.032876+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:46.032876+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12291, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.029317+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:46.029317+0000', 'last_peered': '2024-09-11T05:08:46.029317+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:46.029317+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12771, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032865+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:46.032865+0000', 'last_peered': '2024-09-11T05:08:46.032865+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:46.032865+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12933, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032853+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:46.032853+0000', 'last_peered': '2024-09-11T05:08:46.032853+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:46.032853+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12793, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.032746+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:46.032746+0000', 'last_peered': '2024-09-11T05:08:46.032746+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:46.032746+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-11T05:09:01.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12590", 'reported_seq': 12706, 'reported_epoch': 48, 'state': 'peering', 'last_fresh': '2024-09-11T05:07:50.442739+0000', 'last_change': '2024-09-11T05:08:57.093579+0000', 'last_active': '2024-09-11T05:08:57.093579+0000', 'last_peered': '2024-09-11T05:07:50.442739+0000', 'last_clean': '2024-09-11T04:55:07.995693+0000', 'last_became_active': '2024-09-11T04:56:20.585620+0000', 'last_became_peered': '2024-09-11T04:56:20.585620+0000', 'last_unstale': '2024-09-11T05:07:50.442739+0000', 'last_undegraded': '2024-09-11T04:56:20.277378+0000', 'last_fullsized': '2024-09-11T04:56:20.228093+0000', 'mapping_epoch': 48, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12588, 'num_write_kb': 50364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [1, 3, 2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-11T05:09:01.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:01.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12552, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:46.029135+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:46.029135+0000', 'last_peered': '2024-09-11T05:08:46.029135+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:46.029135+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:01.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:09:01.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 13224, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:57.410453+0000', 'last_change': '2024-09-11T05:08:56.392290+0000', 'last_active': '2024-09-11T05:08:57.410453+0000', 'last_peered': '2024-09-11T05:08:57.410453+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:08:57.410453+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 175, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2138}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:09:01.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:01.832 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:01.833 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg peering","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15912/56 objects degraded (28414.286%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":868,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14,"data_bytes":94304,"bytes_used":2873094144,"bytes_avail":383673962496,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":15912,"degraded_total":56,"degraded_ratio":284.14285714285717},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:01.847 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:09:01.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:02.158 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:02.159 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:02.171 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-11T05:09:01.942674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805788,"kb_used_data":1620,"kb_used_omap":14,"kb_used_meta":2804081,"kb_avail":374681572,"statfs":{"total":386547056640,"available":383673929728,"internally_reserved":0,"allocated":1658880,"data_stored":654762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2871379510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-848,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001389"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12638,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070280+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:58.070280+0000","last_peered":"2024-09-11T05:08:58.070280+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:58.070280+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12889,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070801+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:57.070801+0000","last_peered":"2024-09-11T05:08:57.070801+0000","last_clean":"2024-09-11T05:08:57.070801+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:57.070801+0000","last_undegraded":"2024-09-11T05:08:57.070801+0000","last_fullsized":"2024-09-11T05:08:57.070801+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12295,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076527+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:58.076527+0000","last_peered":"2024-09-11T05:08:58.076527+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:58.076527+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13956,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070608+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:57.070608+0000","last_peered":"2024-09-11T05:08:57.070608+0000","last_clean":"2024-09-11T05:08:57.070608+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:57.070608+0000","last_undegraded":"2024-09-11T05:08:57.070608+0000","last_fullsized":"2024-09-11T05:08:57.070608+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17146,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.069897+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:58.069897+0000","last_peered":"2024-09-11T05:08:58.069897+0000","last_clean":"2024-09-11T05:08:58.069897+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:58.069897+0000","last_undegraded":"2024-09-11T05:08:58.069897+0000","last_fullsized":"2024-09-11T05:08:58.069897+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12775,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070134+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:58.070134+0000","last_peered":"2024-09-11T05:08:58.070134+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:58.070134+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17486,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.067825+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:57.067825+0000","last_peered":"2024-09-11T05:08:57.067825+0000","last_clean":"2024-09-11T05:08:57.067825+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:57.067825+0000","last_undegraded":"2024-09-11T05:08:57.067825+0000","last_fullsized":"2024-09-11T05:08:57.067825+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17708,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.071464+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:57.071464+0000","last_peered":"2024-09-11T05:08:57.071464+0000","last_clean":"2024-09-11T05:08:57.071464+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:57.071464+0000","last_undegraded":"2024-09-11T05:08:57.071464+0000","last_fullsized":"2024-09-11T05:08:57.071464+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12937,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.069997+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:58.069997+0000","last_peered":"2024-09-11T05:08:58.069997+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:58.069997+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17302,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.076888+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:58.076888+0000","last_peered":"2024-09-11T05:08:58.076888+0000","last_clean":"2024-09-11T05:08:58.076888+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:58.076888+0000","last_undegraded":"2024-09-11T05:08:58.076888+0000","last_fullsized":"2024-09-11T05:08:58.076888+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12797,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070031+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:58.070031+0000","last_peered":"2024-09-11T05:08:58.070031+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:58.070031+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17013,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070909+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:57.070909+0000","last_peered":"2024-09-11T05:08:57.070909+0000","last_clean":"2024-09-11T05:08:57.070909+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:57.070909+0000","last_undegraded":"2024-09-11T05:08:57.070909+0000","last_fullsized":"2024-09-11T05:08:57.070909+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17527,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:08:57.070636+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:57.070636+0000","last_peered":"2024-09-11T05:08:57.070636+0000","last_clean":"2024-09-11T05:08:57.070636+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:57.070636+0000","last_undegraded":"2024-09-11T05:08:57.070636+0000","last_fullsized":"2024-09-11T05:08:57.070636+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12706,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-11T05:07:50.442739+0000","last_change":"2024-09-11T05:08:57.093579+0000","last_active":"2024-09-11T05:08:57.093579+0000","last_peered":"2024-09-11T05:07:50.442739+0000","last_clean":"2024-09-11T04:55:07.995693+0000","last_became_active":"2024-09-11T04:56:20.585620+0000","last_became_peered":"2024-09-11T04:56:20.585620+0000","last_unstale":"2024-09-11T05:07:50.442739+0000","last_undegraded":"2024-09-11T04:56:20.277378+0000","last_fullsized":"2024-09-11T04:56:20.228093+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12556,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076566+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:58.076566+0000","last_peered":"2024-09-11T05:08:58.076566+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:58.076566+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":13224,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:57.410453+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:08:57.410453+0000","last_peered":"2024-09-11T05:08:57.410453+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:08:57.410453+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94304,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":402097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744296,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627544,"statfs":{"total":96636764160,"available":95874605056,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721576,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650264,"statfs":{"total":96636764160,"available":95897870336,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617912,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753928,"statfs":{"total":96636764160,"available":96004022272,"internally_reserved":0,"allocated":434176,"data_stored":188022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":632286930},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722004,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649836,"statfs":{"total":96636764160,"available":95897432064,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:05.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:05.486 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:05.498 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-11T05:09:03.943098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793348,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2791601,"kb_avail":374694012,"statfs":{"total":386547056640,"available":383686668288,"internally_reserved":0,"allocated":1691648,"data_stored":668028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2858599990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12588,"num_write_kb":-50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1029,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001460"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12638,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070280+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:58.070280+0000","last_peered":"2024-09-11T05:08:58.070280+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:58.070280+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12891,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074381+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:58.074381+0000","last_peered":"2024-09-11T05:08:58.074381+0000","last_clean":"2024-09-11T05:08:58.074381+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:58.074381+0000","last_undegraded":"2024-09-11T05:08:58.074381+0000","last_fullsized":"2024-09-11T05:08:58.074381+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12295,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076527+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:58.076527+0000","last_peered":"2024-09-11T05:08:58.076527+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:58.076527+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13958,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073734+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:58.073734+0000","last_peered":"2024-09-11T05:08:58.073734+0000","last_clean":"2024-09-11T05:08:58.073734+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:58.073734+0000","last_undegraded":"2024-09-11T05:08:58.073734+0000","last_fullsized":"2024-09-11T05:08:58.073734+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17146,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.069897+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:58.069897+0000","last_peered":"2024-09-11T05:08:58.069897+0000","last_clean":"2024-09-11T05:08:58.069897+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:58.069897+0000","last_undegraded":"2024-09-11T05:08:58.069897+0000","last_fullsized":"2024-09-11T05:08:58.069897+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12775,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070134+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:58.070134+0000","last_peered":"2024-09-11T05:08:58.070134+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:58.070134+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17488,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073924+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:58.073924+0000","last_peered":"2024-09-11T05:08:58.073924+0000","last_clean":"2024-09-11T05:08:58.073924+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:58.073924+0000","last_undegraded":"2024-09-11T05:08:58.073924+0000","last_fullsized":"2024-09-11T05:08:58.073924+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17710,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074334+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:58.074334+0000","last_peered":"2024-09-11T05:08:58.074334+0000","last_clean":"2024-09-11T05:08:58.074334+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:58.074334+0000","last_undegraded":"2024-09-11T05:08:58.074334+0000","last_fullsized":"2024-09-11T05:08:58.074334+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12937,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.069997+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:58.069997+0000","last_peered":"2024-09-11T05:08:58.069997+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:58.069997+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17302,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.076888+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:58.076888+0000","last_peered":"2024-09-11T05:08:58.076888+0000","last_clean":"2024-09-11T05:08:58.076888+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:58.076888+0000","last_undegraded":"2024-09-11T05:08:58.076888+0000","last_fullsized":"2024-09-11T05:08:58.076888+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12797,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070031+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:58.070031+0000","last_peered":"2024-09-11T05:08:58.070031+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:58.070031+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17015,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074503+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:58.074503+0000","last_peered":"2024-09-11T05:08:58.074503+0000","last_clean":"2024-09-11T05:08:58.074503+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:58.074503+0000","last_undegraded":"2024-09-11T05:08:58.074503+0000","last_fullsized":"2024-09-11T05:08:58.074503+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17529,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073845+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:58.073845+0000","last_peered":"2024-09-11T05:08:58.073845+0000","last_clean":"2024-09-11T05:08:58.073845+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:58.073845+0000","last_undegraded":"2024-09-11T05:08:58.073845+0000","last_fullsized":"2024-09-11T05:08:58.073845+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12725,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.817214+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:08:58.817214+0000","last_peered":"2024-09-11T05:08:58.817214+0000","last_clean":"2024-09-11T05:08:58.817214+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:08:58.817214+0000","last_undegraded":"2024-09-11T05:08:58.817214+0000","last_fullsized":"2024-09-11T05:08:58.817214+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12556,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076566+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:58.076566+0000","last_peered":"2024-09-11T05:08:58.076566+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:58.076566+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":14936,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:02.413770+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:09:02.413770+0000","last_peered":"2024-09-11T05:09:02.413770+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:09:02.413770+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744296,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627544,"statfs":{"total":96636764160,"available":95874605056,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721576,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650264,"statfs":{"total":96636764160,"available":95897870336,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605448,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93766392,"statfs":{"total":96636764160,"available":96016785408,"internally_reserved":0,"allocated":450560,"data_stored":194655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619507410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:05.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:05.818 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:05.831 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-11T05:09:03.943098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793348,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2791601,"kb_avail":374694012,"statfs":{"total":386547056640,"available":383686668288,"internally_reserved":0,"allocated":1691648,"data_stored":668028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2858599990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12588,"num_write_kb":-50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1029,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001460"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12638,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070280+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:58.070280+0000","last_peered":"2024-09-11T05:08:58.070280+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:58.070280+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12891,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074381+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:58.074381+0000","last_peered":"2024-09-11T05:08:58.074381+0000","last_clean":"2024-09-11T05:08:58.074381+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:58.074381+0000","last_undegraded":"2024-09-11T05:08:58.074381+0000","last_fullsized":"2024-09-11T05:08:58.074381+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12295,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076527+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:58.076527+0000","last_peered":"2024-09-11T05:08:58.076527+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:58.076527+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13958,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073734+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:58.073734+0000","last_peered":"2024-09-11T05:08:58.073734+0000","last_clean":"2024-09-11T05:08:58.073734+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:58.073734+0000","last_undegraded":"2024-09-11T05:08:58.073734+0000","last_fullsized":"2024-09-11T05:08:58.073734+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17146,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.069897+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:58.069897+0000","last_peered":"2024-09-11T05:08:58.069897+0000","last_clean":"2024-09-11T05:08:58.069897+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:58.069897+0000","last_undegraded":"2024-09-11T05:08:58.069897+0000","last_fullsized":"2024-09-11T05:08:58.069897+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12775,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070134+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:58.070134+0000","last_peered":"2024-09-11T05:08:58.070134+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:58.070134+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17488,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073924+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:58.073924+0000","last_peered":"2024-09-11T05:08:58.073924+0000","last_clean":"2024-09-11T05:08:58.073924+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:58.073924+0000","last_undegraded":"2024-09-11T05:08:58.073924+0000","last_fullsized":"2024-09-11T05:08:58.073924+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17710,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074334+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:58.074334+0000","last_peered":"2024-09-11T05:08:58.074334+0000","last_clean":"2024-09-11T05:08:58.074334+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:58.074334+0000","last_undegraded":"2024-09-11T05:08:58.074334+0000","last_fullsized":"2024-09-11T05:08:58.074334+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12937,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.069997+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:58.069997+0000","last_peered":"2024-09-11T05:08:58.069997+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:58.069997+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17302,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.076888+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:58.076888+0000","last_peered":"2024-09-11T05:08:58.076888+0000","last_clean":"2024-09-11T05:08:58.076888+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:58.076888+0000","last_undegraded":"2024-09-11T05:08:58.076888+0000","last_fullsized":"2024-09-11T05:08:58.076888+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12797,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070031+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:58.070031+0000","last_peered":"2024-09-11T05:08:58.070031+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:58.070031+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17015,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074503+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:58.074503+0000","last_peered":"2024-09-11T05:08:58.074503+0000","last_clean":"2024-09-11T05:08:58.074503+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:58.074503+0000","last_undegraded":"2024-09-11T05:08:58.074503+0000","last_fullsized":"2024-09-11T05:08:58.074503+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17529,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073845+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:58.073845+0000","last_peered":"2024-09-11T05:08:58.073845+0000","last_clean":"2024-09-11T05:08:58.073845+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:58.073845+0000","last_undegraded":"2024-09-11T05:08:58.073845+0000","last_fullsized":"2024-09-11T05:08:58.073845+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12725,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.817214+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:08:58.817214+0000","last_peered":"2024-09-11T05:08:58.817214+0000","last_clean":"2024-09-11T05:08:58.817214+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:08:58.817214+0000","last_undegraded":"2024-09-11T05:08:58.817214+0000","last_fullsized":"2024-09-11T05:08:58.817214+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12556,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076566+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:58.076566+0000","last_peered":"2024-09-11T05:08:58.076566+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:58.076566+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":14936,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:02.413770+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:09:02.413770+0000","last_peered":"2024-09-11T05:09:02.413770+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:09:02.413770+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744296,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627544,"statfs":{"total":96636764160,"available":95874605056,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721576,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650264,"statfs":{"total":96636764160,"available":95897870336,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605448,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93766392,"statfs":{"total":96636764160,"available":96016785408,"internally_reserved":0,"allocated":450560,"data_stored":194655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619507410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:05.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:05.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12638, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.070280+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:58.070280+0000', 'last_peered': '2024-09-11T05:08:58.070280+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:58.070280+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:05.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:09:05.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12295, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.076527+0000', 'last_change': '2024-09-11T04:56:20.638789+0000', 'last_active': '2024-09-11T05:08:58.076527+0000', 'last_peered': '2024-09-11T05:08:58.076527+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:08:58.076527+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:05.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12775, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.070134+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:58.070134+0000', 'last_peered': '2024-09-11T05:08:58.070134+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:58.070134+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12937, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.069997+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:58.069997+0000', 'last_peered': '2024-09-11T05:08:58.069997+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:58.069997+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12797, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.070031+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:58.070031+0000', 'last_peered': '2024-09-11T05:08:58.070031+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:58.070031+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12556, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.076566+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:08:58.076566+0000', 'last_peered': '2024-09-11T05:08:58.076566+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:08:58.076566+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:09:05.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 14936, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:02.413770+0000', 'last_change': '2024-09-11T05:08:56.392290+0000', 'last_active': '2024-09-11T05:09:02.413770+0000', 'last_peered': '2024-09-11T05:09:02.413770+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:09:02.413770+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1030, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:09:05.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15057/48 objects degraded (31368.750%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":873,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":80832,"bytes_used":2860388352,"bytes_avail":383686668288,"bytes_total":386547056640,"degraded_objects":15057,"degraded_total":48,"degraded_ratio":313.6875},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:06.250 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:09:06.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:06.572 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:06.585 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-11T05:09:05.943431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793348,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2791601,"kb_avail":374694012,"statfs":{"total":386547056640,"available":383686668288,"internally_reserved":0,"allocated":1691648,"data_stored":668028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2858599990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12588,"num_write_kb":-50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1029,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.878681"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12638,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070280+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:58.070280+0000","last_peered":"2024-09-11T05:08:58.070280+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:58.070280+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12891,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074381+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:58.074381+0000","last_peered":"2024-09-11T05:08:58.074381+0000","last_clean":"2024-09-11T05:08:58.074381+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:58.074381+0000","last_undegraded":"2024-09-11T05:08:58.074381+0000","last_fullsized":"2024-09-11T05:08:58.074381+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12295,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076527+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:58.076527+0000","last_peered":"2024-09-11T05:08:58.076527+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:58.076527+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13958,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073734+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:58.073734+0000","last_peered":"2024-09-11T05:08:58.073734+0000","last_clean":"2024-09-11T05:08:58.073734+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:58.073734+0000","last_undegraded":"2024-09-11T05:08:58.073734+0000","last_fullsized":"2024-09-11T05:08:58.073734+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17146,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.069897+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:58.069897+0000","last_peered":"2024-09-11T05:08:58.069897+0000","last_clean":"2024-09-11T05:08:58.069897+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:58.069897+0000","last_undegraded":"2024-09-11T05:08:58.069897+0000","last_fullsized":"2024-09-11T05:08:58.069897+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12776,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.071329+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:58.071329+0000","last_peered":"2024-09-11T05:08:58.071329+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:58.071329+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17488,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073924+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:58.073924+0000","last_peered":"2024-09-11T05:08:58.073924+0000","last_clean":"2024-09-11T05:08:58.073924+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:58.073924+0000","last_undegraded":"2024-09-11T05:08:58.073924+0000","last_fullsized":"2024-09-11T05:08:58.073924+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17710,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074334+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:58.074334+0000","last_peered":"2024-09-11T05:08:58.074334+0000","last_clean":"2024-09-11T05:08:58.074334+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:58.074334+0000","last_undegraded":"2024-09-11T05:08:58.074334+0000","last_fullsized":"2024-09-11T05:08:58.074334+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12937,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.069997+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:58.069997+0000","last_peered":"2024-09-11T05:08:58.069997+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:58.069997+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17302,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.076888+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:58.076888+0000","last_peered":"2024-09-11T05:08:58.076888+0000","last_clean":"2024-09-11T05:08:58.076888+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:58.076888+0000","last_undegraded":"2024-09-11T05:08:58.076888+0000","last_fullsized":"2024-09-11T05:08:58.076888+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12797,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070031+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:58.070031+0000","last_peered":"2024-09-11T05:08:58.070031+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:58.070031+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17015,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074503+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:58.074503+0000","last_peered":"2024-09-11T05:08:58.074503+0000","last_clean":"2024-09-11T05:08:58.074503+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:58.074503+0000","last_undegraded":"2024-09-11T05:08:58.074503+0000","last_fullsized":"2024-09-11T05:08:58.074503+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17529,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073845+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:58.073845+0000","last_peered":"2024-09-11T05:08:58.073845+0000","last_clean":"2024-09-11T05:08:58.073845+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:58.073845+0000","last_undegraded":"2024-09-11T05:08:58.073845+0000","last_fullsized":"2024-09-11T05:08:58.073845+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12725,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.817214+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:08:58.817214+0000","last_peered":"2024-09-11T05:08:58.817214+0000","last_clean":"2024-09-11T05:08:58.817214+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:08:58.817214+0000","last_undegraded":"2024-09-11T05:08:58.817214+0000","last_fullsized":"2024-09-11T05:08:58.817214+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12556,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076566+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:58.076566+0000","last_peered":"2024-09-11T05:08:58.076566+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:58.076566+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":14936,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:02.413770+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:09:02.413770+0000","last_peered":"2024-09-11T05:09:02.413770+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:09:02.413770+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744296,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627544,"statfs":{"total":96636764160,"available":95874605056,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721576,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650264,"statfs":{"total":96636764160,"available":95897870336,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605448,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93766392,"statfs":{"total":96636764160,"available":96016785408,"internally_reserved":0,"allocated":450560,"data_stored":194655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619507410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:09.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:09.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:09.930 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:09.942 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-11T05:09:09.147024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801668,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2799921,"kb_avail":374685692,"statfs":{"total":386547056640,"available":383678148608,"internally_reserved":0,"allocated":1691648,"data_stored":668028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2867119670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12588,"num_write_kb":-50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1878,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.077472"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12638,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070280+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:58.070280+0000","last_peered":"2024-09-11T05:08:58.070280+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:58.070280+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12891,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074381+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:58.074381+0000","last_peered":"2024-09-11T05:08:58.074381+0000","last_clean":"2024-09-11T05:08:58.074381+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:58.074381+0000","last_undegraded":"2024-09-11T05:08:58.074381+0000","last_fullsized":"2024-09-11T05:08:58.074381+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12295,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076527+0000","last_change":"2024-09-11T04:56:20.638789+0000","last_active":"2024-09-11T05:08:58.076527+0000","last_peered":"2024-09-11T05:08:58.076527+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:08:58.076527+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13958,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073734+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:58.073734+0000","last_peered":"2024-09-11T05:08:58.073734+0000","last_clean":"2024-09-11T05:08:58.073734+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:58.073734+0000","last_undegraded":"2024-09-11T05:08:58.073734+0000","last_fullsized":"2024-09-11T05:08:58.073734+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17146,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.069897+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:58.069897+0000","last_peered":"2024-09-11T05:08:58.069897+0000","last_clean":"2024-09-11T05:08:58.069897+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:58.069897+0000","last_undegraded":"2024-09-11T05:08:58.069897+0000","last_fullsized":"2024-09-11T05:08:58.069897+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12776,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.071329+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:58.071329+0000","last_peered":"2024-09-11T05:08:58.071329+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:58.071329+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17488,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073924+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:58.073924+0000","last_peered":"2024-09-11T05:08:58.073924+0000","last_clean":"2024-09-11T05:08:58.073924+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:58.073924+0000","last_undegraded":"2024-09-11T05:08:58.073924+0000","last_fullsized":"2024-09-11T05:08:58.073924+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17710,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074334+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:58.074334+0000","last_peered":"2024-09-11T05:08:58.074334+0000","last_clean":"2024-09-11T05:08:58.074334+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:58.074334+0000","last_undegraded":"2024-09-11T05:08:58.074334+0000","last_fullsized":"2024-09-11T05:08:58.074334+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12937,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.069997+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:58.069997+0000","last_peered":"2024-09-11T05:08:58.069997+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:58.069997+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17302,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.076888+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:08:58.076888+0000","last_peered":"2024-09-11T05:08:58.076888+0000","last_clean":"2024-09-11T05:08:58.076888+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:08:58.076888+0000","last_undegraded":"2024-09-11T05:08:58.076888+0000","last_fullsized":"2024-09-11T05:08:58.076888+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12797,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070031+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:58.070031+0000","last_peered":"2024-09-11T05:08:58.070031+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:58.070031+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17015,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074503+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:58.074503+0000","last_peered":"2024-09-11T05:08:58.074503+0000","last_clean":"2024-09-11T05:08:58.074503+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:58.074503+0000","last_undegraded":"2024-09-11T05:08:58.074503+0000","last_fullsized":"2024-09-11T05:08:58.074503+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17529,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073845+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:58.073845+0000","last_peered":"2024-09-11T05:08:58.073845+0000","last_clean":"2024-09-11T05:08:58.073845+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:58.073845+0000","last_undegraded":"2024-09-11T05:08:58.073845+0000","last_fullsized":"2024-09-11T05:08:58.073845+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12725,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.817214+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:08:58.817214+0000","last_peered":"2024-09-11T05:08:58.817214+0000","last_clean":"2024-09-11T05:08:58.817214+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:08:58.817214+0000","last_undegraded":"2024-09-11T05:08:58.817214+0000","last_fullsized":"2024-09-11T05:08:58.817214+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12556,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.076566+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:08:58.076566+0000","last_peered":"2024-09-11T05:08:58.076566+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:08:58.076566+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":17032,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:07.412958+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:09:07.412958+0000","last_peered":"2024-09-11T05:09:07.412958+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:09:07.412958+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744296,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627544,"statfs":{"total":96636764160,"available":95874605056,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721576,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650264,"statfs":{"total":96636764160,"available":95897870336,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613768,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93758072,"statfs":{"total":96636764160,"available":96008265728,"internally_reserved":0,"allocated":450560,"data_stored":194655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628027090},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:09.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:10.306 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:10.307 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:10.320 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-11T05:09:10.155126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801676,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":2799921,"kb_avail":374685684,"statfs":{"total":386547056640,"available":383678140416,"internally_reserved":0,"allocated":1699840,"data_stored":671318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2867119670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12588,"num_write_kb":-50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.212452"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12638,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070280+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:58.070280+0000","last_peered":"2024-09-11T05:08:58.070280+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:58.070280+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12891,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074381+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:58.074381+0000","last_peered":"2024-09-11T05:08:58.074381+0000","last_clean":"2024-09-11T05:08:58.074381+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:58.074381+0000","last_undegraded":"2024-09-11T05:08:58.074381+0000","last_fullsized":"2024-09-11T05:08:58.074381+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12790,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.071648+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:10.071648+0000","last_peered":"2024-09-11T05:09:10.071648+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:10.071648+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13958,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073734+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:58.073734+0000","last_peered":"2024-09-11T05:08:58.073734+0000","last_clean":"2024-09-11T05:08:58.073734+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:58.073734+0000","last_undegraded":"2024-09-11T05:08:58.073734+0000","last_fullsized":"2024-09-11T05:08:58.073734+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17146,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.069897+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:58.069897+0000","last_peered":"2024-09-11T05:08:58.069897+0000","last_clean":"2024-09-11T05:08:58.069897+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:58.069897+0000","last_undegraded":"2024-09-11T05:08:58.069897+0000","last_fullsized":"2024-09-11T05:08:58.069897+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12776,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.071329+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:58.071329+0000","last_peered":"2024-09-11T05:08:58.071329+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:58.071329+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17488,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073924+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:58.073924+0000","last_peered":"2024-09-11T05:08:58.073924+0000","last_clean":"2024-09-11T05:08:58.073924+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:58.073924+0000","last_undegraded":"2024-09-11T05:08:58.073924+0000","last_fullsized":"2024-09-11T05:08:58.073924+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17710,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074334+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:58.074334+0000","last_peered":"2024-09-11T05:08:58.074334+0000","last_clean":"2024-09-11T05:08:58.074334+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:58.074334+0000","last_undegraded":"2024-09-11T05:08:58.074334+0000","last_fullsized":"2024-09-11T05:08:58.074334+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12937,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.069997+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:58.069997+0000","last_peered":"2024-09-11T05:08:58.069997+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:58.069997+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17304,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-11T05:09:09.150037+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:09.150037+0000","last_peered":"2024-09-11T05:09:09.150037+0000","last_clean":"2024-09-11T05:09:09.150037+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:09.150037+0000","last_undegraded":"2024-09-11T05:09:09.150037+0000","last_fullsized":"2024-09-11T05:09:09.150037+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12797,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070031+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:58.070031+0000","last_peered":"2024-09-11T05:08:58.070031+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:58.070031+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17015,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074503+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:58.074503+0000","last_peered":"2024-09-11T05:08:58.074503+0000","last_clean":"2024-09-11T05:08:58.074503+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:58.074503+0000","last_undegraded":"2024-09-11T05:08:58.074503+0000","last_fullsized":"2024-09-11T05:08:58.074503+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17529,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073845+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:58.073845+0000","last_peered":"2024-09-11T05:08:58.073845+0000","last_clean":"2024-09-11T05:08:58.073845+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:58.073845+0000","last_undegraded":"2024-09-11T05:08:58.073845+0000","last_fullsized":"2024-09-11T05:08:58.073845+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12725,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.817214+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:08:58.817214+0000","last_peered":"2024-09-11T05:08:58.817214+0000","last_clean":"2024-09-11T05:08:58.817214+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:08:58.817214+0000","last_undegraded":"2024-09-11T05:08:58.817214+0000","last_fullsized":"2024-09-11T05:08:58.817214+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:09.148693+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:09.148693+0000","last_peered":"2024-09-11T05:09:09.148693+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:09.148693+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":17032,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:07.412958+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:09:07.412958+0000","last_peered":"2024-09-11T05:09:07.412958+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:09:07.412958+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744296,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627544,"statfs":{"total":96636764160,"available":95874605056,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721584,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650256,"statfs":{"total":96636764160,"available":95897862144,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613768,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93758072,"statfs":{"total":96636764160,"available":96008265728,"internally_reserved":0,"allocated":450560,"data_stored":194655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628027090},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:10.321 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:10.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12638, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.070280+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:08:58.070280+0000', 'last_peered': '2024-09-11T05:08:58.070280+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:08:58.070280+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:10.321 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:09:10.321 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12790, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.071648+0000', 'last_change': '2024-09-11T05:09:08.301934+0000', 'last_active': '2024-09-11T05:09:10.071648+0000', 'last_peered': '2024-09-11T05:09:10.071648+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:09:10.071648+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 256, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1965}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12776, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.071329+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:08:58.071329+0000', 'last_peered': '2024-09-11T05:08:58.071329+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:08:58.071329+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12937, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.069997+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:08:58.069997+0000', 'last_peered': '2024-09-11T05:08:58.069997+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:08:58.069997+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12797, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:08:58.070031+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:08:58.070031+0000', 'last_peered': '2024-09-11T05:08:58.070031+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:08:58.070031+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12558, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:09.148693+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:09:09.148693+0000', 'last_peered': '2024-09-11T05:09:09.148693+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:09.148693+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-11T05:09:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12775", 'reported_seq': 17032, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:07.412958+0000', 'last_change': '2024-09-11T05:08:56.392290+0000', 'last_active': '2024-09-11T05:09:07.412958+0000', 'last_peered': '2024-09-11T05:09:07.412958+0000', 'last_clean': '2024-09-11T04:55:07.784589+0000', 'last_became_active': '2024-09-11T04:56:20.639248+0000', 'last_became_peered': '2024-09-11T04:56:20.639248+0000', 'last_unstale': '2024-09-11T05:09:07.412958+0000', 'last_undegraded': '2024-09-11T04:56:20.287423+0000', 'last_fullsized': '2024-09-11T04:56:20.223275+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12773, 'num_write_kb': 51100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2078, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -235}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-11T05:09:10.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:10.828 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:10.828 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14009/48 objects degraded (29185.417%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":877,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":80832,"bytes_used":2868908032,"bytes_avail":383678148608,"bytes_total":386547056640,"degraded_objects":14009,"degraded_total":48,"degraded_ratio":291.85416666666669,"recovering_objects_per_sec":187,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1878,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:10.841 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:09:10.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:11.167 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:11.180 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-11T05:09:10.155126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801676,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":2799921,"kb_avail":374685684,"statfs":{"total":386547056640,"available":383678140416,"internally_reserved":0,"allocated":1699840,"data_stored":671318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2867119670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12588,"num_write_kb":-50364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.212452"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12638,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070280+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:08:58.070280+0000","last_peered":"2024-09-11T05:08:58.070280+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:08:58.070280+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12891,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074381+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:08:58.074381+0000","last_peered":"2024-09-11T05:08:58.074381+0000","last_clean":"2024-09-11T05:08:58.074381+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:08:58.074381+0000","last_undegraded":"2024-09-11T05:08:58.074381+0000","last_fullsized":"2024-09-11T05:08:58.074381+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12790,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.071648+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:10.071648+0000","last_peered":"2024-09-11T05:09:10.071648+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:10.071648+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13958,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073734+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:08:58.073734+0000","last_peered":"2024-09-11T05:08:58.073734+0000","last_clean":"2024-09-11T05:08:58.073734+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:08:58.073734+0000","last_undegraded":"2024-09-11T05:08:58.073734+0000","last_fullsized":"2024-09-11T05:08:58.073734+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17146,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.069897+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:08:58.069897+0000","last_peered":"2024-09-11T05:08:58.069897+0000","last_clean":"2024-09-11T05:08:58.069897+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:08:58.069897+0000","last_undegraded":"2024-09-11T05:08:58.069897+0000","last_fullsized":"2024-09-11T05:08:58.069897+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12776,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.071329+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:08:58.071329+0000","last_peered":"2024-09-11T05:08:58.071329+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:08:58.071329+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17488,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073924+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:08:58.073924+0000","last_peered":"2024-09-11T05:08:58.073924+0000","last_clean":"2024-09-11T05:08:58.073924+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:08:58.073924+0000","last_undegraded":"2024-09-11T05:08:58.073924+0000","last_fullsized":"2024-09-11T05:08:58.073924+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17710,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074334+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:08:58.074334+0000","last_peered":"2024-09-11T05:08:58.074334+0000","last_clean":"2024-09-11T05:08:58.074334+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:08:58.074334+0000","last_undegraded":"2024-09-11T05:08:58.074334+0000","last_fullsized":"2024-09-11T05:08:58.074334+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12937,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.069997+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:08:58.069997+0000","last_peered":"2024-09-11T05:08:58.069997+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:08:58.069997+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17304,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-11T05:09:09.150037+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:09.150037+0000","last_peered":"2024-09-11T05:09:09.150037+0000","last_clean":"2024-09-11T05:09:09.150037+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:09.150037+0000","last_undegraded":"2024-09-11T05:09:09.150037+0000","last_fullsized":"2024-09-11T05:09:09.150037+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12797,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:08:58.070031+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:08:58.070031+0000","last_peered":"2024-09-11T05:08:58.070031+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:08:58.070031+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17015,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.074503+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:08:58.074503+0000","last_peered":"2024-09-11T05:08:58.074503+0000","last_clean":"2024-09-11T05:08:58.074503+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:08:58.074503+0000","last_undegraded":"2024-09-11T05:08:58.074503+0000","last_fullsized":"2024-09-11T05:08:58.074503+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17529,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.073845+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:08:58.073845+0000","last_peered":"2024-09-11T05:08:58.073845+0000","last_clean":"2024-09-11T05:08:58.073845+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:08:58.073845+0000","last_undegraded":"2024-09-11T05:08:58.073845+0000","last_fullsized":"2024-09-11T05:08:58.073845+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12725,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T05:08:58.817214+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:08:58.817214+0000","last_peered":"2024-09-11T05:08:58.817214+0000","last_clean":"2024-09-11T05:08:58.817214+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:08:58.817214+0000","last_undegraded":"2024-09-11T05:08:58.817214+0000","last_fullsized":"2024-09-11T05:08:58.817214+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:09.148693+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:09.148693+0000","last_peered":"2024-09-11T05:09:09.148693+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:09.148693+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":17032,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:07.412958+0000","last_change":"2024-09-11T05:08:56.392290+0000","last_active":"2024-09-11T05:09:07.412958+0000","last_peered":"2024-09-11T05:09:07.412958+0000","last_clean":"2024-09-11T04:55:07.784589+0000","last_became_active":"2024-09-11T04:56:20.639248+0000","last_became_peered":"2024-09-11T04:56:20.639248+0000","last_unstale":"2024-09-11T05:09:07.412958+0000","last_undegraded":"2024-09-11T04:56:20.287423+0000","last_fullsized":"2024-09-11T04:56:20.223275+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80832,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98515,"num_write_kb":351733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744296,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627544,"statfs":{"total":96636764160,"available":95874605056,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721584,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650256,"statfs":{"total":96636764160,"available":95897862144,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613768,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93758072,"statfs":{"total":96636764160,"available":96008265728,"internally_reserved":0,"allocated":450560,"data_stored":194655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628027090},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722028,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649812,"statfs":{"total":96636764160,"available":95897407488,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:14.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:14.493 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:14.494 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:14.505 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-11T05:09:13.944771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793400,"kb_used_data":1704,"kb_used_omap":14,"kb_used_meta":2791601,"kb_avail":374693960,"statfs":{"total":386547056640,"available":383686615040,"internally_reserved":0,"allocated":1744896,"data_stored":686899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2858599990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12773,"num_write_kb":-51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001340"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12895,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160304+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:10.160304+0000","last_peered":"2024-09-11T05:09:10.160304+0000","last_clean":"2024-09-11T05:09:10.160304+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:10.160304+0000","last_undegraded":"2024-09-11T05:09:10.160304+0000","last_fullsized":"2024-09-11T05:09:10.160304+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12790,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.071648+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:10.071648+0000","last_peered":"2024-09-11T05:09:10.071648+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:10.071648+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13962,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160000+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:10.160000+0000","last_peered":"2024-09-11T05:09:10.160000+0000","last_clean":"2024-09-11T05:09:10.160000+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:10.160000+0000","last_undegraded":"2024-09-11T05:09:10.160000+0000","last_fullsized":"2024-09-11T05:09:10.160000+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12780,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.158325+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:09:10.158325+0000","last_peered":"2024-09-11T05:09:10.158325+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:10.158325+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17492,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.157426+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:10.157426+0000","last_peered":"2024-09-11T05:09:10.157426+0000","last_clean":"2024-09-11T05:09:10.157426+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:10.157426+0000","last_undegraded":"2024-09-11T05:09:10.157426+0000","last_fullsized":"2024-09-11T05:09:10.157426+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17714,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160603+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:10.160603+0000","last_peered":"2024-09-11T05:09:10.160603+0000","last_clean":"2024-09-11T05:09:10.160603+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:10.160603+0000","last_undegraded":"2024-09-11T05:09:10.160603+0000","last_fullsized":"2024-09-11T05:09:10.160603+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17304,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-11T05:09:09.150037+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:09.150037+0000","last_peered":"2024-09-11T05:09:09.150037+0000","last_clean":"2024-09-11T05:09:09.150037+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:09.150037+0000","last_undegraded":"2024-09-11T05:09:09.150037+0000","last_fullsized":"2024-09-11T05:09:09.150037+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17019,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160231+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:10.160231+0000","last_peered":"2024-09-11T05:09:10.160231+0000","last_clean":"2024-09-11T05:09:10.160231+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:10.160231+0000","last_undegraded":"2024-09-11T05:09:10.160231+0000","last_fullsized":"2024-09-11T05:09:10.160231+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17533,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160037+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:10.160037+0000","last_peered":"2024-09-11T05:09:10.160037+0000","last_clean":"2024-09-11T05:09:10.160037+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:10.160037+0000","last_undegraded":"2024-09-11T05:09:10.160037+0000","last_fullsized":"2024-09-11T05:09:10.160037+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12729,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160105+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:10.160105+0000","last_peered":"2024-09-11T05:09:10.160105+0000","last_clean":"2024-09-11T05:09:10.160105+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:10.160105+0000","last_undegraded":"2024-09-11T05:09:10.160105+0000","last_fullsized":"2024-09-11T05:09:10.160105+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:09.148693+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:09.148693+0000","last_peered":"2024-09-11T05:09:09.148693+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:09.148693+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12923,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.916233+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:10.916233+0000","last_peered":"2024-09-11T05:09:10.916233+0000","last_clean":"2024-09-11T05:09:10.916233+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:10.916233+0000","last_undegraded":"2024-09-11T05:09:10.916233+0000","last_fullsized":"2024-09-11T05:09:10.916233+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721584,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650256,"statfs":{"total":96636764160,"available":95897862144,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605460,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93766380,"statfs":{"total":96636764160,"available":96016773120,"internally_reserved":0,"allocated":462848,"data_stored":197118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619507410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:14.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:14.811 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:14.824 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-11T05:09:13.944771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793400,"kb_used_data":1704,"kb_used_omap":14,"kb_used_meta":2791601,"kb_avail":374693960,"statfs":{"total":386547056640,"available":383686615040,"internally_reserved":0,"allocated":1744896,"data_stored":686899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2858599990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12773,"num_write_kb":-51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001340"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12895,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160304+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:10.160304+0000","last_peered":"2024-09-11T05:09:10.160304+0000","last_clean":"2024-09-11T05:09:10.160304+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:10.160304+0000","last_undegraded":"2024-09-11T05:09:10.160304+0000","last_fullsized":"2024-09-11T05:09:10.160304+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12790,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.071648+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:10.071648+0000","last_peered":"2024-09-11T05:09:10.071648+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:10.071648+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13962,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160000+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:10.160000+0000","last_peered":"2024-09-11T05:09:10.160000+0000","last_clean":"2024-09-11T05:09:10.160000+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:10.160000+0000","last_undegraded":"2024-09-11T05:09:10.160000+0000","last_fullsized":"2024-09-11T05:09:10.160000+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12780,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.158325+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:09:10.158325+0000","last_peered":"2024-09-11T05:09:10.158325+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:10.158325+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17492,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.157426+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:10.157426+0000","last_peered":"2024-09-11T05:09:10.157426+0000","last_clean":"2024-09-11T05:09:10.157426+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:10.157426+0000","last_undegraded":"2024-09-11T05:09:10.157426+0000","last_fullsized":"2024-09-11T05:09:10.157426+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17714,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160603+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:10.160603+0000","last_peered":"2024-09-11T05:09:10.160603+0000","last_clean":"2024-09-11T05:09:10.160603+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:10.160603+0000","last_undegraded":"2024-09-11T05:09:10.160603+0000","last_fullsized":"2024-09-11T05:09:10.160603+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17304,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-11T05:09:09.150037+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:09.150037+0000","last_peered":"2024-09-11T05:09:09.150037+0000","last_clean":"2024-09-11T05:09:09.150037+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:09.150037+0000","last_undegraded":"2024-09-11T05:09:09.150037+0000","last_fullsized":"2024-09-11T05:09:09.150037+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17019,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160231+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:10.160231+0000","last_peered":"2024-09-11T05:09:10.160231+0000","last_clean":"2024-09-11T05:09:10.160231+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:10.160231+0000","last_undegraded":"2024-09-11T05:09:10.160231+0000","last_fullsized":"2024-09-11T05:09:10.160231+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17533,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160037+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:10.160037+0000","last_peered":"2024-09-11T05:09:10.160037+0000","last_clean":"2024-09-11T05:09:10.160037+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:10.160037+0000","last_undegraded":"2024-09-11T05:09:10.160037+0000","last_fullsized":"2024-09-11T05:09:10.160037+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12729,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160105+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:10.160105+0000","last_peered":"2024-09-11T05:09:10.160105+0000","last_clean":"2024-09-11T05:09:10.160105+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:10.160105+0000","last_undegraded":"2024-09-11T05:09:10.160105+0000","last_fullsized":"2024-09-11T05:09:10.160105+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:09.148693+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:09.148693+0000","last_peered":"2024-09-11T05:09:09.148693+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:09.148693+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12923,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.916233+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:10.916233+0000","last_peered":"2024-09-11T05:09:10.916233+0000","last_clean":"2024-09-11T05:09:10.916233+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:10.916233+0000","last_undegraded":"2024-09-11T05:09:10.916233+0000","last_fullsized":"2024-09-11T05:09:10.916233+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721584,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650256,"statfs":{"total":96636764160,"available":95897862144,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605460,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93766380,"statfs":{"total":96636764160,"available":96016773120,"internally_reserved":0,"allocated":462848,"data_stored":197118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619507410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12643, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.157625+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:10.157625+0000', 'last_peered': '2024-09-11T05:09:10.157625+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:10.157625+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:14.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 12790, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.071648+0000', 'last_change': '2024-09-11T05:09:08.301934+0000', 'last_active': '2024-09-11T05:09:10.071648+0000', 'last_peered': '2024-09-11T05:09:10.071648+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:09:10.071648+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 256, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1965}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12780, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.158325+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:09:10.158325+0000', 'last_peered': '2024-09-11T05:09:10.158325+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:09:10.158325+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12941, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.156701+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:09:10.156701+0000', 'last_peered': '2024-09-11T05:09:10.156701+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:10.156701+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12801, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.157840+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:10.157840+0000', 'last_peered': '2024-09-11T05:09:10.157840+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:10.157840+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:14.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12558, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:09.148693+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:09:09.148693+0000', 'last_peered': '2024-09-11T05:09:09.148693+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:09.148693+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:14.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:15.260 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:15.260 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13526/40 objects degraded (33815.000%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":882,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10,"data_bytes":67360,"bytes_used":2860441600,"bytes_avail":383686615040,"bytes_total":386547056640,"degraded_objects":13526,"degraded_total":40,"degraded_ratio":338.14999999999998},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:15.276 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:09:15.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:15.612 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:15.612 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:15.625 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-11T05:09:13.944771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793400,"kb_used_data":1704,"kb_used_omap":14,"kb_used_meta":2791601,"kb_avail":374693960,"statfs":{"total":386547056640,"available":383686615040,"internally_reserved":0,"allocated":1744896,"data_stored":686899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2858599990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12773,"num_write_kb":-51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001340"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12895,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160304+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:10.160304+0000","last_peered":"2024-09-11T05:09:10.160304+0000","last_clean":"2024-09-11T05:09:10.160304+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:10.160304+0000","last_undegraded":"2024-09-11T05:09:10.160304+0000","last_fullsized":"2024-09-11T05:09:10.160304+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":12790,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.071648+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:10.071648+0000","last_peered":"2024-09-11T05:09:10.071648+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:10.071648+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13962,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160000+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:10.160000+0000","last_peered":"2024-09-11T05:09:10.160000+0000","last_clean":"2024-09-11T05:09:10.160000+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:10.160000+0000","last_undegraded":"2024-09-11T05:09:10.160000+0000","last_fullsized":"2024-09-11T05:09:10.160000+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12780,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.158325+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:09:10.158325+0000","last_peered":"2024-09-11T05:09:10.158325+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:10.158325+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17492,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.157426+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:10.157426+0000","last_peered":"2024-09-11T05:09:10.157426+0000","last_clean":"2024-09-11T05:09:10.157426+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:10.157426+0000","last_undegraded":"2024-09-11T05:09:10.157426+0000","last_fullsized":"2024-09-11T05:09:10.157426+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17714,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160603+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:10.160603+0000","last_peered":"2024-09-11T05:09:10.160603+0000","last_clean":"2024-09-11T05:09:10.160603+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:10.160603+0000","last_undegraded":"2024-09-11T05:09:10.160603+0000","last_fullsized":"2024-09-11T05:09:10.160603+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17304,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-11T05:09:09.150037+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:09.150037+0000","last_peered":"2024-09-11T05:09:09.150037+0000","last_clean":"2024-09-11T05:09:09.150037+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:09.150037+0000","last_undegraded":"2024-09-11T05:09:09.150037+0000","last_fullsized":"2024-09-11T05:09:09.150037+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17019,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160231+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:10.160231+0000","last_peered":"2024-09-11T05:09:10.160231+0000","last_clean":"2024-09-11T05:09:10.160231+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:10.160231+0000","last_undegraded":"2024-09-11T05:09:10.160231+0000","last_fullsized":"2024-09-11T05:09:10.160231+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17533,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160037+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:10.160037+0000","last_peered":"2024-09-11T05:09:10.160037+0000","last_clean":"2024-09-11T05:09:10.160037+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:10.160037+0000","last_undegraded":"2024-09-11T05:09:10.160037+0000","last_fullsized":"2024-09-11T05:09:10.160037+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12729,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160105+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:10.160105+0000","last_peered":"2024-09-11T05:09:10.160105+0000","last_clean":"2024-09-11T05:09:10.160105+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:10.160105+0000","last_undegraded":"2024-09-11T05:09:10.160105+0000","last_fullsized":"2024-09-11T05:09:10.160105+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:09.148693+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:09.148693+0000","last_peered":"2024-09-11T05:09:09.148693+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:09.148693+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12923,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.916233+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:10.916233+0000","last_peered":"2024-09-11T05:09:10.916233+0000","last_clean":"2024-09-11T05:09:10.916233+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:10.916233+0000","last_undegraded":"2024-09-11T05:09:10.916233+0000","last_fullsized":"2024-09-11T05:09:10.916233+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721584,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650256,"statfs":{"total":96636764160,"available":95897862144,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605460,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93766380,"statfs":{"total":96636764160,"available":96016773120,"internally_reserved":0,"allocated":462848,"data_stored":197118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":619507410},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:18.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:19.084 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:19.085 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:19.100 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-11T05:09:17.945527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801728,"kb_used_data":1712,"kb_used_omap":14,"kb_used_meta":2799921,"kb_avail":374685632,"statfs":{"total":386547056640,"available":383678087168,"internally_reserved":0,"allocated":1753088,"data_stored":690168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2867119670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12773,"num_write_kb":-51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.798502"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12895,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160304+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:10.160304+0000","last_peered":"2024-09-11T05:09:10.160304+0000","last_clean":"2024-09-11T05:09:10.160304+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:10.160304+0000","last_undegraded":"2024-09-11T05:09:10.160304+0000","last_fullsized":"2024-09-11T05:09:10.160304+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":14504,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:15.073624+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:15.073624+0000","last_peered":"2024-09-11T05:09:15.073624+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:15.073624+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13962,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160000+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:10.160000+0000","last_peered":"2024-09-11T05:09:10.160000+0000","last_clean":"2024-09-11T05:09:10.160000+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:10.160000+0000","last_undegraded":"2024-09-11T05:09:10.160000+0000","last_fullsized":"2024-09-11T05:09:10.160000+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12780,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.158325+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:09:10.158325+0000","last_peered":"2024-09-11T05:09:10.158325+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:10.158325+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17492,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.157426+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:10.157426+0000","last_peered":"2024-09-11T05:09:10.157426+0000","last_clean":"2024-09-11T05:09:10.157426+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:10.157426+0000","last_undegraded":"2024-09-11T05:09:10.157426+0000","last_fullsized":"2024-09-11T05:09:10.157426+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17715,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160663+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:10.160663+0000","last_peered":"2024-09-11T05:09:10.160663+0000","last_clean":"2024-09-11T05:09:10.160663+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:10.160663+0000","last_undegraded":"2024-09-11T05:09:10.160663+0000","last_fullsized":"2024-09-11T05:09:10.160663+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17306,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.164721+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:10.164721+0000","last_peered":"2024-09-11T05:09:10.164721+0000","last_clean":"2024-09-11T05:09:10.164721+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:10.164721+0000","last_undegraded":"2024-09-11T05:09:10.164721+0000","last_fullsized":"2024-09-11T05:09:10.164721+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17019,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160231+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:10.160231+0000","last_peered":"2024-09-11T05:09:10.160231+0000","last_clean":"2024-09-11T05:09:10.160231+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:10.160231+0000","last_undegraded":"2024-09-11T05:09:10.160231+0000","last_fullsized":"2024-09-11T05:09:10.160231+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17533,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160037+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:10.160037+0000","last_peered":"2024-09-11T05:09:10.160037+0000","last_clean":"2024-09-11T05:09:10.160037+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:10.160037+0000","last_undegraded":"2024-09-11T05:09:10.160037+0000","last_fullsized":"2024-09-11T05:09:10.160037+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12729,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160105+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:10.160105+0000","last_peered":"2024-09-11T05:09:10.160105+0000","last_clean":"2024-09-11T05:09:10.160105+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:10.160105+0000","last_undegraded":"2024-09-11T05:09:10.160105+0000","last_fullsized":"2024-09-11T05:09:10.160105+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12560,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.162037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:10.162037+0000","last_peered":"2024-09-11T05:09:10.162037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:10.162037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12923,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.916233+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:10.916233+0000","last_peered":"2024-09-11T05:09:10.916233+0000","last_clean":"2024-09-11T05:09:10.916233+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:10.916233+0000","last_undegraded":"2024-09-11T05:09:10.916233+0000","last_fullsized":"2024-09-11T05:09:10.916233+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721592,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650248,"statfs":{"total":96636764160,"available":95897853952,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613780,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93758060,"statfs":{"total":96636764160,"available":96008253440,"internally_reserved":0,"allocated":462848,"data_stored":197118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628027090},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:19.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:19.414 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:19.428 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-11T05:09:17.945527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801728,"kb_used_data":1712,"kb_used_omap":14,"kb_used_meta":2799921,"kb_avail":374685632,"statfs":{"total":386547056640,"available":383678087168,"internally_reserved":0,"allocated":1753088,"data_stored":690168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2867119670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12773,"num_write_kb":-51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.798502"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12895,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160304+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:10.160304+0000","last_peered":"2024-09-11T05:09:10.160304+0000","last_clean":"2024-09-11T05:09:10.160304+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:10.160304+0000","last_undegraded":"2024-09-11T05:09:10.160304+0000","last_fullsized":"2024-09-11T05:09:10.160304+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":14504,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:15.073624+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:15.073624+0000","last_peered":"2024-09-11T05:09:15.073624+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:15.073624+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13962,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160000+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:10.160000+0000","last_peered":"2024-09-11T05:09:10.160000+0000","last_clean":"2024-09-11T05:09:10.160000+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:10.160000+0000","last_undegraded":"2024-09-11T05:09:10.160000+0000","last_fullsized":"2024-09-11T05:09:10.160000+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12780,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.158325+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:09:10.158325+0000","last_peered":"2024-09-11T05:09:10.158325+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:10.158325+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17492,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.157426+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:10.157426+0000","last_peered":"2024-09-11T05:09:10.157426+0000","last_clean":"2024-09-11T05:09:10.157426+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:10.157426+0000","last_undegraded":"2024-09-11T05:09:10.157426+0000","last_fullsized":"2024-09-11T05:09:10.157426+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17715,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160663+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:10.160663+0000","last_peered":"2024-09-11T05:09:10.160663+0000","last_clean":"2024-09-11T05:09:10.160663+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:10.160663+0000","last_undegraded":"2024-09-11T05:09:10.160663+0000","last_fullsized":"2024-09-11T05:09:10.160663+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17306,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.164721+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:10.164721+0000","last_peered":"2024-09-11T05:09:10.164721+0000","last_clean":"2024-09-11T05:09:10.164721+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:10.164721+0000","last_undegraded":"2024-09-11T05:09:10.164721+0000","last_fullsized":"2024-09-11T05:09:10.164721+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17019,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160231+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:10.160231+0000","last_peered":"2024-09-11T05:09:10.160231+0000","last_clean":"2024-09-11T05:09:10.160231+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:10.160231+0000","last_undegraded":"2024-09-11T05:09:10.160231+0000","last_fullsized":"2024-09-11T05:09:10.160231+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17533,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160037+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:10.160037+0000","last_peered":"2024-09-11T05:09:10.160037+0000","last_clean":"2024-09-11T05:09:10.160037+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:10.160037+0000","last_undegraded":"2024-09-11T05:09:10.160037+0000","last_fullsized":"2024-09-11T05:09:10.160037+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12729,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160105+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:10.160105+0000","last_peered":"2024-09-11T05:09:10.160105+0000","last_clean":"2024-09-11T05:09:10.160105+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:10.160105+0000","last_undegraded":"2024-09-11T05:09:10.160105+0000","last_fullsized":"2024-09-11T05:09:10.160105+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12560,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.162037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:10.162037+0000","last_peered":"2024-09-11T05:09:10.162037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:10.162037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12923,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.916233+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:10.916233+0000","last_peered":"2024-09-11T05:09:10.916233+0000","last_clean":"2024-09-11T05:09:10.916233+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:10.916233+0000","last_undegraded":"2024-09-11T05:09:10.916233+0000","last_fullsized":"2024-09-11T05:09:10.916233+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721592,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650248,"statfs":{"total":96636764160,"available":95897853952,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613780,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93758060,"statfs":{"total":96636764160,"available":96008253440,"internally_reserved":0,"allocated":462848,"data_stored":197118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628027090},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:19.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:19.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12643, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.157625+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:10.157625+0000', 'last_peered': '2024-09-11T05:09:10.157625+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:10.157625+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:19.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:09:19.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 14504, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:15.073624+0000', 'last_change': '2024-09-11T05:09:08.301934+0000', 'last_active': '2024-09-11T05:09:15.073624+0000', 'last_peered': '2024-09-11T05:09:15.073624+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:09:15.073624+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1112, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1109}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:19.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:09:19.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12780, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.158325+0000', 'last_change': '2024-09-11T04:56:20.600412+0000', 'last_active': '2024-09-11T05:09:10.158325+0000', 'last_peered': '2024-09-11T05:09:10.158325+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:09:10.158325+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:19.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:19.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12941, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.156701+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:09:10.156701+0000', 'last_peered': '2024-09-11T05:09:10.156701+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:10.156701+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:19.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:19.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12801, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.157840+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:10.157840+0000', 'last_peered': '2024-09-11T05:09:10.157840+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:10.157840+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:19.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:19.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12560, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.162037+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:09:10.162037+0000', 'last_peered': '2024-09-11T05:09:10.162037+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:10.162037+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:19.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:19.853 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:19.853 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12670/40 objects degraded (31675.000%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":886,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10,"data_bytes":67360,"bytes_used":2868969472,"bytes_avail":383678087168,"bytes_total":386547056640,"degraded_objects":12670,"degraded_total":40,"degraded_ratio":316.75},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:19.867 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:09:19.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:20.259 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:20.260 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:20.272 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-09-11T05:09:19.945859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801728,"kb_used_data":1712,"kb_used_omap":14,"kb_used_meta":2799921,"kb_avail":374685632,"statfs":{"total":386547056640,"available":383678087168,"internally_reserved":0,"allocated":1753088,"data_stored":690168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2867119670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12773,"num_write_kb":-51100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001755"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12895,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160304+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:10.160304+0000","last_peered":"2024-09-11T05:09:10.160304+0000","last_clean":"2024-09-11T05:09:10.160304+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:10.160304+0000","last_undegraded":"2024-09-11T05:09:10.160304+0000","last_fullsized":"2024-09-11T05:09:10.160304+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":14504,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:15.073624+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:15.073624+0000","last_peered":"2024-09-11T05:09:15.073624+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:15.073624+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13962,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160000+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:10.160000+0000","last_peered":"2024-09-11T05:09:10.160000+0000","last_clean":"2024-09-11T05:09:10.160000+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:10.160000+0000","last_undegraded":"2024-09-11T05:09:10.160000+0000","last_fullsized":"2024-09-11T05:09:10.160000+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12780,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.158325+0000","last_change":"2024-09-11T04:56:20.600412+0000","last_active":"2024-09-11T05:09:10.158325+0000","last_peered":"2024-09-11T05:09:10.158325+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:10.158325+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17492,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.157426+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:10.157426+0000","last_peered":"2024-09-11T05:09:10.157426+0000","last_clean":"2024-09-11T05:09:10.157426+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:10.157426+0000","last_undegraded":"2024-09-11T05:09:10.157426+0000","last_fullsized":"2024-09-11T05:09:10.157426+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17715,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160663+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:10.160663+0000","last_peered":"2024-09-11T05:09:10.160663+0000","last_clean":"2024-09-11T05:09:10.160663+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:10.160663+0000","last_undegraded":"2024-09-11T05:09:10.160663+0000","last_fullsized":"2024-09-11T05:09:10.160663+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17306,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.164721+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:10.164721+0000","last_peered":"2024-09-11T05:09:10.164721+0000","last_clean":"2024-09-11T05:09:10.164721+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:10.164721+0000","last_undegraded":"2024-09-11T05:09:10.164721+0000","last_fullsized":"2024-09-11T05:09:10.164721+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17019,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160231+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:10.160231+0000","last_peered":"2024-09-11T05:09:10.160231+0000","last_clean":"2024-09-11T05:09:10.160231+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:10.160231+0000","last_undegraded":"2024-09-11T05:09:10.160231+0000","last_fullsized":"2024-09-11T05:09:10.160231+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17533,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160037+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:10.160037+0000","last_peered":"2024-09-11T05:09:10.160037+0000","last_clean":"2024-09-11T05:09:10.160037+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:10.160037+0000","last_undegraded":"2024-09-11T05:09:10.160037+0000","last_fullsized":"2024-09-11T05:09:10.160037+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12729,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.160105+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:10.160105+0000","last_peered":"2024-09-11T05:09:10.160105+0000","last_clean":"2024-09-11T05:09:10.160105+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:10.160105+0000","last_undegraded":"2024-09-11T05:09:10.160105+0000","last_fullsized":"2024-09-11T05:09:10.160105+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12560,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.162037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:10.162037+0000","last_peered":"2024-09-11T05:09:10.162037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:10.162037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12923,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.916233+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:10.916233+0000","last_peered":"2024-09-11T05:09:10.916233+0000","last_clean":"2024-09-11T05:09:10.916233+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:10.916233+0000","last_undegraded":"2024-09-11T05:09:10.916233+0000","last_fullsized":"2024-09-11T05:09:10.916233+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721592,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650248,"statfs":{"total":96636764160,"available":95897853952,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613780,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93758060,"statfs":{"total":96636764160,"available":96008253440,"internally_reserved":0,"allocated":462848,"data_stored":197118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":628027090},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:23.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:23.729 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-11T05:09:22.196502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811468,"kb_used_data":1720,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675892,"statfs":{"total":386547056640,"available":383668113408,"internally_reserved":0,"allocated":1761280,"data_stored":693421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.251730"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12897,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194297+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:21.194297+0000","last_peered":"2024-09-11T05:09:21.194297+0000","last_clean":"2024-09-11T05:09:21.194297+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:21.194297+0000","last_undegraded":"2024-09-11T05:09:21.194297+0000","last_fullsized":"2024-09-11T05:09:21.194297+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16610,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:20.072194+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:20.072194+0000","last_peered":"2024-09-11T05:09:20.072194+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:20.072194+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":57,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-56}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13964,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193715+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:21.193715+0000","last_peered":"2024-09-11T05:09:21.193715+0000","last_clean":"2024-09-11T05:09:21.193715+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:21.193715+0000","last_undegraded":"2024-09-11T05:09:21.193715+0000","last_fullsized":"2024-09-11T05:09:21.193715+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12793,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:20.403102+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:20.403102+0000","last_peered":"2024-09-11T05:09:20.403102+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:20.403102+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2288}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17492,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.157426+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:10.157426+0000","last_peered":"2024-09-11T05:09:10.157426+0000","last_clean":"2024-09-11T05:09:10.157426+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:10.157426+0000","last_undegraded":"2024-09-11T05:09:10.157426+0000","last_fullsized":"2024-09-11T05:09:10.157426+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17717,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193994+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:21.193994+0000","last_peered":"2024-09-11T05:09:21.193994+0000","last_clean":"2024-09-11T05:09:21.193994+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:21.193994+0000","last_undegraded":"2024-09-11T05:09:21.193994+0000","last_fullsized":"2024-09-11T05:09:21.193994+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17306,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.164721+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:10.164721+0000","last_peered":"2024-09-11T05:09:10.164721+0000","last_clean":"2024-09-11T05:09:10.164721+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:10.164721+0000","last_undegraded":"2024-09-11T05:09:10.164721+0000","last_fullsized":"2024-09-11T05:09:10.164721+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17021,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194401+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:21.194401+0000","last_peered":"2024-09-11T05:09:21.194401+0000","last_clean":"2024-09-11T05:09:21.194401+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:21.194401+0000","last_undegraded":"2024-09-11T05:09:21.194401+0000","last_fullsized":"2024-09-11T05:09:21.194401+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17535,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193830+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:21.193830+0000","last_peered":"2024-09-11T05:09:21.193830+0000","last_clean":"2024-09-11T05:09:21.193830+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:21.193830+0000","last_undegraded":"2024-09-11T05:09:21.193830+0000","last_fullsized":"2024-09-11T05:09:21.193830+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12731,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193874+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:21.193874+0000","last_peered":"2024-09-11T05:09:21.193874+0000","last_clean":"2024-09-11T05:09:21.193874+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:21.193874+0000","last_undegraded":"2024-09-11T05:09:21.193874+0000","last_fullsized":"2024-09-11T05:09:21.193874+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12560,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.162037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:10.162037+0000","last_peered":"2024-09-11T05:09:10.162037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:10.162037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194096+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:21.194096+0000","last_peered":"2024-09-11T05:09:21.194096+0000","last_clean":"2024-09-11T05:09:21.194096+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:21.194096+0000","last_undegraded":"2024-09-11T05:09:21.194096+0000","last_fullsized":"2024-09-11T05:09:21.194096+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721592,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650248,"statfs":{"total":96636764160,"available":95897853952,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623520,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748320,"statfs":{"total":96636764160,"available":95998279680,"internally_reserved":0,"allocated":471040,"data_stored":200371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722044,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649796,"statfs":{"total":96636764160,"available":95897391104,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:23.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:24.057 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-11T05:09:23.946548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811484,"kb_used_data":1736,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675876,"statfs":{"total":386547056640,"available":383668097024,"internally_reserved":0,"allocated":1777664,"data_stored":699906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001347"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12897,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194297+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:21.194297+0000","last_peered":"2024-09-11T05:09:21.194297+0000","last_clean":"2024-09-11T05:09:21.194297+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:21.194297+0000","last_undegraded":"2024-09-11T05:09:21.194297+0000","last_fullsized":"2024-09-11T05:09:21.194297+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16610,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:20.072194+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:20.072194+0000","last_peered":"2024-09-11T05:09:20.072194+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:20.072194+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":57,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-56}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13964,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193715+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:21.193715+0000","last_peered":"2024-09-11T05:09:21.193715+0000","last_clean":"2024-09-11T05:09:21.193715+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:21.193715+0000","last_undegraded":"2024-09-11T05:09:21.193715+0000","last_fullsized":"2024-09-11T05:09:21.193715+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12793,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:20.403102+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:20.403102+0000","last_peered":"2024-09-11T05:09:20.403102+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:20.403102+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2288}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17496,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201993+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:22.201993+0000","last_peered":"2024-09-11T05:09:22.201993+0000","last_clean":"2024-09-11T05:09:22.201993+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:22.201993+0000","last_undegraded":"2024-09-11T05:09:22.201993+0000","last_fullsized":"2024-09-11T05:09:22.201993+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17717,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193994+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:21.193994+0000","last_peered":"2024-09-11T05:09:21.193994+0000","last_clean":"2024-09-11T05:09:21.193994+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:21.193994+0000","last_undegraded":"2024-09-11T05:09:21.193994+0000","last_fullsized":"2024-09-11T05:09:21.193994+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17306,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.164721+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:10.164721+0000","last_peered":"2024-09-11T05:09:10.164721+0000","last_clean":"2024-09-11T05:09:10.164721+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:10.164721+0000","last_undegraded":"2024-09-11T05:09:10.164721+0000","last_fullsized":"2024-09-11T05:09:10.164721+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17021,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194401+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:21.194401+0000","last_peered":"2024-09-11T05:09:21.194401+0000","last_clean":"2024-09-11T05:09:21.194401+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:21.194401+0000","last_undegraded":"2024-09-11T05:09:21.194401+0000","last_fullsized":"2024-09-11T05:09:21.194401+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17535,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193830+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:21.193830+0000","last_peered":"2024-09-11T05:09:21.193830+0000","last_clean":"2024-09-11T05:09:21.193830+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:21.193830+0000","last_undegraded":"2024-09-11T05:09:21.193830+0000","last_fullsized":"2024-09-11T05:09:21.193830+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12731,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193874+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:21.193874+0000","last_peered":"2024-09-11T05:09:21.193874+0000","last_clean":"2024-09-11T05:09:21.193874+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:21.193874+0000","last_undegraded":"2024-09-11T05:09:21.193874+0000","last_fullsized":"2024-09-11T05:09:21.193874+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12560,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.162037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:10.162037+0000","last_peered":"2024-09-11T05:09:10.162037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:10.162037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194096+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:21.194096+0000","last_peered":"2024-09-11T05:09:21.194096+0000","last_clean":"2024-09-11T05:09:21.194096+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:21.194096+0000","last_undegraded":"2024-09-11T05:09:21.194096+0000","last_fullsized":"2024-09-11T05:09:21.194096+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721592,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650248,"statfs":{"total":96636764160,"available":95897853952,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623520,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748320,"statfs":{"total":96636764160,"available":95998279680,"internally_reserved":0,"allocated":471040,"data_stored":200371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:24.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:24.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12643, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.157625+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:10.157625+0000', 'last_peered': '2024-09-11T05:09:10.157625+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:10.157625+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12160", 'reported_seq': 16610, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:20.072194+0000', 'last_change': '2024-09-11T05:09:08.301934+0000', 'last_active': '2024-09-11T05:09:20.072194+0000', 'last_peered': '2024-09-11T05:09:20.072194+0000', 'last_clean': '2024-09-11T04:55:08.455708+0000', 'last_became_active': '2024-09-11T04:56:20.624432+0000', 'last_became_peered': '2024-09-11T04:56:20.624432+0000', 'last_unstale': '2024-09-11T05:09:20.072194+0000', 'last_undegraded': '2024-09-11T04:56:20.301085+0000', 'last_fullsized': '2024-09-11T04:56:20.278578+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6736, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 57, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12159, 'num_write_kb': 48631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2165, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -56}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 12793, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:20.403102+0000', 'last_change': '2024-09-11T05:09:20.357434+0000', 'last_active': '2024-09-11T05:09:20.403102+0000', 'last_peered': '2024-09-11T05:09:20.403102+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:09:20.403102+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2288, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2288}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12941, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.156701+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:09:10.156701+0000', 'last_peered': '2024-09-11T05:09:10.156701+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:10.156701+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12801, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.157840+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:10.157840+0000', 'last_peered': '2024-09-11T05:09:10.157840+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:10.157840+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:24.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12560, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:10.162037+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:09:10.162037+0000', 'last_peered': '2024-09-11T05:09:10.162037+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:10.162037+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:24.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:24.519 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11611/40 objects degraded (29027.500%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":891,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":10,"data_bytes":67360,"bytes_used":2878959616,"bytes_avail":383668097024,"bytes_total":386547056640,"degraded_objects":11611,"degraded_total":40,"degraded_ratio":290.27499999999998,"recovering_objects_per_sec":132,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:24.532 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:09:24.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:24.854 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:24.867 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-11T05:09:23.946548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811484,"kb_used_data":1736,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675876,"statfs":{"total":386547056640,"available":383668097024,"internally_reserved":0,"allocated":1777664,"data_stored":699906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001347"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12643,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157625+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:10.157625+0000","last_peered":"2024-09-11T05:09:10.157625+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:10.157625+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12897,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194297+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:21.194297+0000","last_peered":"2024-09-11T05:09:21.194297+0000","last_clean":"2024-09-11T05:09:21.194297+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:21.194297+0000","last_undegraded":"2024-09-11T05:09:21.194297+0000","last_fullsized":"2024-09-11T05:09:21.194297+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16610,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:20.072194+0000","last_change":"2024-09-11T05:09:08.301934+0000","last_active":"2024-09-11T05:09:20.072194+0000","last_peered":"2024-09-11T05:09:20.072194+0000","last_clean":"2024-09-11T04:55:08.455708+0000","last_became_active":"2024-09-11T04:56:20.624432+0000","last_became_peered":"2024-09-11T04:56:20.624432+0000","last_unstale":"2024-09-11T05:09:20.072194+0000","last_undegraded":"2024-09-11T04:56:20.301085+0000","last_fullsized":"2024-09-11T04:56:20.278578+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":57,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12159,"num_write_kb":48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-56}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13964,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193715+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:21.193715+0000","last_peered":"2024-09-11T05:09:21.193715+0000","last_clean":"2024-09-11T05:09:21.193715+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:21.193715+0000","last_undegraded":"2024-09-11T05:09:21.193715+0000","last_fullsized":"2024-09-11T05:09:21.193715+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17150,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.156153+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:10.156153+0000","last_peered":"2024-09-11T05:09:10.156153+0000","last_clean":"2024-09-11T05:09:10.156153+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:10.156153+0000","last_undegraded":"2024-09-11T05:09:10.156153+0000","last_fullsized":"2024-09-11T05:09:10.156153+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":12793,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:20.403102+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:20.403102+0000","last_peered":"2024-09-11T05:09:20.403102+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:20.403102+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2288}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17496,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201993+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:22.201993+0000","last_peered":"2024-09-11T05:09:22.201993+0000","last_clean":"2024-09-11T05:09:22.201993+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:22.201993+0000","last_undegraded":"2024-09-11T05:09:22.201993+0000","last_fullsized":"2024-09-11T05:09:22.201993+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17717,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193994+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:21.193994+0000","last_peered":"2024-09-11T05:09:21.193994+0000","last_clean":"2024-09-11T05:09:21.193994+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:21.193994+0000","last_undegraded":"2024-09-11T05:09:21.193994+0000","last_fullsized":"2024-09-11T05:09:21.193994+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12941,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.156701+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:10.156701+0000","last_peered":"2024-09-11T05:09:10.156701+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:10.156701+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17306,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-11T05:09:10.164721+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:10.164721+0000","last_peered":"2024-09-11T05:09:10.164721+0000","last_clean":"2024-09-11T05:09:10.164721+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:10.164721+0000","last_undegraded":"2024-09-11T05:09:10.164721+0000","last_fullsized":"2024-09-11T05:09:10.164721+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12801,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.157840+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:10.157840+0000","last_peered":"2024-09-11T05:09:10.157840+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:10.157840+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17021,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194401+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:21.194401+0000","last_peered":"2024-09-11T05:09:21.194401+0000","last_clean":"2024-09-11T05:09:21.194401+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:21.194401+0000","last_undegraded":"2024-09-11T05:09:21.194401+0000","last_fullsized":"2024-09-11T05:09:21.194401+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17535,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193830+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:21.193830+0000","last_peered":"2024-09-11T05:09:21.193830+0000","last_clean":"2024-09-11T05:09:21.193830+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:21.193830+0000","last_undegraded":"2024-09-11T05:09:21.193830+0000","last_fullsized":"2024-09-11T05:09:21.193830+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12731,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.193874+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:21.193874+0000","last_peered":"2024-09-11T05:09:21.193874+0000","last_clean":"2024-09-11T05:09:21.193874+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:21.193874+0000","last_undegraded":"2024-09-11T05:09:21.193874+0000","last_fullsized":"2024-09-11T05:09:21.193874+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12560,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:10.162037+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:10.162037+0000","last_peered":"2024-09-11T05:09:10.162037+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:10.162037+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-11T05:09:21.194096+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:21.194096+0000","last_peered":"2024-09-11T05:09:21.194096+0000","last_clean":"2024-09-11T05:09:21.194096+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:21.194096+0000","last_undegraded":"2024-09-11T05:09:21.194096+0000","last_fullsized":"2024-09-11T05:09:21.194096+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67360,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85742,"num_write_kb":300633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744312,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627528,"statfs":{"total":96636764160,"available":95874588672,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721592,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650248,"statfs":{"total":96636764160,"available":95897853952,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623520,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748320,"statfs":{"total":96636764160,"available":95998279680,"internally_reserved":0,"allocated":471040,"data_stored":200371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:27.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:28.200 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:28.200 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:28.213 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-11T05:09:27.947197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819840,"kb_used_data":1776,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667520,"statfs":{"total":386547056640,"available":383659540480,"internally_reserved":0,"allocated":1818624,"data_stored":716108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12159,"num_write_kb":-48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001338"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12647,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198808+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:22.198808+0000","last_peered":"2024-09-11T05:09:22.198808+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:22.198808+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12899,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202048+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:22.202048+0000","last_peered":"2024-09-11T05:09:22.202048+0000","last_clean":"2024-09-11T05:09:22.202048+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:22.202048+0000","last_undegraded":"2024-09-11T05:09:22.202048+0000","last_fullsized":"2024-09-11T05:09:22.202048+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16745,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:23.176199+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:23.176199+0000","last_peered":"2024-09-11T05:09:23.176199+0000","last_clean":"2024-09-11T05:09:23.176199+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:23.176199+0000","last_undegraded":"2024-09-11T05:09:23.176199+0000","last_fullsized":"2024-09-11T05:09:23.176199+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13966,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202086+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:22.202086+0000","last_peered":"2024-09-11T05:09:22.202086+0000","last_clean":"2024-09-11T05:09:22.202086+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:22.202086+0000","last_undegraded":"2024-09-11T05:09:22.202086+0000","last_fullsized":"2024-09-11T05:09:22.202086+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17154,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.199352+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:22.199352+0000","last_peered":"2024-09-11T05:09:22.199352+0000","last_clean":"2024-09-11T05:09:22.199352+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:22.199352+0000","last_undegraded":"2024-09-11T05:09:22.199352+0000","last_fullsized":"2024-09-11T05:09:22.199352+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":14791,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:25.403696+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:25.403696+0000","last_peered":"2024-09-11T05:09:25.403696+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:25.403696+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17496,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201993+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:22.201993+0000","last_peered":"2024-09-11T05:09:22.201993+0000","last_clean":"2024-09-11T05:09:22.201993+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:22.201993+0000","last_undegraded":"2024-09-11T05:09:22.201993+0000","last_fullsized":"2024-09-11T05:09:22.201993+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17719,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201739+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:22.201739+0000","last_peered":"2024-09-11T05:09:22.201739+0000","last_clean":"2024-09-11T05:09:22.201739+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:22.201739+0000","last_undegraded":"2024-09-11T05:09:22.201739+0000","last_fullsized":"2024-09-11T05:09:22.201739+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12945,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:22.198342+0000","last_peered":"2024-09-11T05:09:22.198342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:22.198342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17310,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.198190+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:22.198190+0000","last_peered":"2024-09-11T05:09:22.198190+0000","last_clean":"2024-09-11T05:09:22.198190+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:22.198190+0000","last_undegraded":"2024-09-11T05:09:22.198190+0000","last_fullsized":"2024-09-11T05:09:22.198190+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12805,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198381+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:22.198381+0000","last_peered":"2024-09-11T05:09:22.198381+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:22.198381+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17023,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202169+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:22.202169+0000","last_peered":"2024-09-11T05:09:22.202169+0000","last_clean":"2024-09-11T05:09:22.202169+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:22.202169+0000","last_undegraded":"2024-09-11T05:09:22.202169+0000","last_fullsized":"2024-09-11T05:09:22.202169+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17537,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202186+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:22.202186+0000","last_peered":"2024-09-11T05:09:22.202186+0000","last_clean":"2024-09-11T05:09:22.202186+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:22.202186+0000","last_undegraded":"2024-09-11T05:09:22.202186+0000","last_fullsized":"2024-09-11T05:09:22.202186+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12733,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201490+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:22.201490+0000","last_peered":"2024-09-11T05:09:22.201490+0000","last_clean":"2024-09-11T05:09:22.201490+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:22.201490+0000","last_undegraded":"2024-09-11T05:09:22.201490+0000","last_fullsized":"2024-09-11T05:09:22.201490+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12564,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.197864+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:22.197864+0000","last_peered":"2024-09-11T05:09:22.197864+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:22.197864+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12927,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201848+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:22.201848+0000","last_peered":"2024-09-11T05:09:22.201848+0000","last_clean":"2024-09-11T05:09:22.201848+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:22.201848+0000","last_undegraded":"2024-09-11T05:09:22.201848+0000","last_fullsized":"2024-09-11T05:09:22.201848+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744328,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627512,"statfs":{"total":96636764160,"available":95874572288,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721608,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650232,"statfs":{"total":96636764160,"available":95897837568,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631844,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739996,"statfs":{"total":96636764160,"available":95989755904,"internally_reserved":0,"allocated":479232,"data_stored":203603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:28.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:28.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:28.523 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:28.535 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-11T05:09:27.947197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819840,"kb_used_data":1776,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667520,"statfs":{"total":386547056640,"available":383659540480,"internally_reserved":0,"allocated":1818624,"data_stored":716108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12159,"num_write_kb":-48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001338"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12647,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198808+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:22.198808+0000","last_peered":"2024-09-11T05:09:22.198808+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:22.198808+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12899,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202048+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:22.202048+0000","last_peered":"2024-09-11T05:09:22.202048+0000","last_clean":"2024-09-11T05:09:22.202048+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:22.202048+0000","last_undegraded":"2024-09-11T05:09:22.202048+0000","last_fullsized":"2024-09-11T05:09:22.202048+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16745,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:23.176199+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:23.176199+0000","last_peered":"2024-09-11T05:09:23.176199+0000","last_clean":"2024-09-11T05:09:23.176199+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:23.176199+0000","last_undegraded":"2024-09-11T05:09:23.176199+0000","last_fullsized":"2024-09-11T05:09:23.176199+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13966,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202086+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:22.202086+0000","last_peered":"2024-09-11T05:09:22.202086+0000","last_clean":"2024-09-11T05:09:22.202086+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:22.202086+0000","last_undegraded":"2024-09-11T05:09:22.202086+0000","last_fullsized":"2024-09-11T05:09:22.202086+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17154,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.199352+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:22.199352+0000","last_peered":"2024-09-11T05:09:22.199352+0000","last_clean":"2024-09-11T05:09:22.199352+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:22.199352+0000","last_undegraded":"2024-09-11T05:09:22.199352+0000","last_fullsized":"2024-09-11T05:09:22.199352+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":14791,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:25.403696+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:25.403696+0000","last_peered":"2024-09-11T05:09:25.403696+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:25.403696+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17496,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201993+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:22.201993+0000","last_peered":"2024-09-11T05:09:22.201993+0000","last_clean":"2024-09-11T05:09:22.201993+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:22.201993+0000","last_undegraded":"2024-09-11T05:09:22.201993+0000","last_fullsized":"2024-09-11T05:09:22.201993+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17719,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201739+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:22.201739+0000","last_peered":"2024-09-11T05:09:22.201739+0000","last_clean":"2024-09-11T05:09:22.201739+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:22.201739+0000","last_undegraded":"2024-09-11T05:09:22.201739+0000","last_fullsized":"2024-09-11T05:09:22.201739+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12945,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:22.198342+0000","last_peered":"2024-09-11T05:09:22.198342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:22.198342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17310,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.198190+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:22.198190+0000","last_peered":"2024-09-11T05:09:22.198190+0000","last_clean":"2024-09-11T05:09:22.198190+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:22.198190+0000","last_undegraded":"2024-09-11T05:09:22.198190+0000","last_fullsized":"2024-09-11T05:09:22.198190+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12805,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198381+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:22.198381+0000","last_peered":"2024-09-11T05:09:22.198381+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:22.198381+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17023,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202169+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:22.202169+0000","last_peered":"2024-09-11T05:09:22.202169+0000","last_clean":"2024-09-11T05:09:22.202169+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:22.202169+0000","last_undegraded":"2024-09-11T05:09:22.202169+0000","last_fullsized":"2024-09-11T05:09:22.202169+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17537,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202186+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:22.202186+0000","last_peered":"2024-09-11T05:09:22.202186+0000","last_clean":"2024-09-11T05:09:22.202186+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:22.202186+0000","last_undegraded":"2024-09-11T05:09:22.202186+0000","last_fullsized":"2024-09-11T05:09:22.202186+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12733,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201490+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:22.201490+0000","last_peered":"2024-09-11T05:09:22.201490+0000","last_clean":"2024-09-11T05:09:22.201490+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:22.201490+0000","last_undegraded":"2024-09-11T05:09:22.201490+0000","last_fullsized":"2024-09-11T05:09:22.201490+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12564,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.197864+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:22.197864+0000","last_peered":"2024-09-11T05:09:22.197864+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:22.197864+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12927,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201848+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:22.201848+0000","last_peered":"2024-09-11T05:09:22.201848+0000","last_clean":"2024-09-11T05:09:22.201848+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:22.201848+0000","last_undegraded":"2024-09-11T05:09:22.201848+0000","last_fullsized":"2024-09-11T05:09:22.201848+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744328,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627512,"statfs":{"total":96636764160,"available":95874572288,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721608,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650232,"statfs":{"total":96636764160,"available":95897837568,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631844,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739996,"statfs":{"total":96636764160,"available":95989755904,"internally_reserved":0,"allocated":479232,"data_stored":203603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:28.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:28.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12647, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:22.198808+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:22.198808+0000', 'last_peered': '2024-09-11T05:09:22.198808+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:22.198808+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:28.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:09:28.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 14791, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:25.403696+0000', 'last_change': '2024-09-11T05:09:20.357434+0000', 'last_active': '2024-09-11T05:09:25.403696+0000', 'last_peered': '2024-09-11T05:09:25.403696+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:09:25.403696+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1030, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:28.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:28.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12945, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:22.198342+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:09:22.198342+0000', 'last_peered': '2024-09-11T05:09:22.198342+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:22.198342+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:28.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:28.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12805, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:22.198381+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:22.198381+0000', 'last_peered': '2024-09-11T05:09:22.198381+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:22.198381+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:28.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:28.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12564, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:22.197864+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:09:22.197864+0000', 'last_peered': '2024-09-11T05:09:22.197864+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:22.197864+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:28.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:28.946 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:28.947 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10557/36 objects degraded (29325.000%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":895,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2887516160,"bytes_avail":383659540480,"bytes_total":386547056640,"degraded_objects":10557,"degraded_total":36,"degraded_ratio":293.25},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:28.960 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:09:28.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:29.277 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:29.278 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:29.289 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-11T05:09:27.947197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819840,"kb_used_data":1776,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667520,"statfs":{"total":386547056640,"available":383659540480,"internally_reserved":0,"allocated":1818624,"data_stored":716108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12159,"num_write_kb":-48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001338"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12647,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198808+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:22.198808+0000","last_peered":"2024-09-11T05:09:22.198808+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:22.198808+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12899,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202048+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:22.202048+0000","last_peered":"2024-09-11T05:09:22.202048+0000","last_clean":"2024-09-11T05:09:22.202048+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:22.202048+0000","last_undegraded":"2024-09-11T05:09:22.202048+0000","last_fullsized":"2024-09-11T05:09:22.202048+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16745,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:23.176199+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:23.176199+0000","last_peered":"2024-09-11T05:09:23.176199+0000","last_clean":"2024-09-11T05:09:23.176199+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:23.176199+0000","last_undegraded":"2024-09-11T05:09:23.176199+0000","last_fullsized":"2024-09-11T05:09:23.176199+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13966,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202086+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:22.202086+0000","last_peered":"2024-09-11T05:09:22.202086+0000","last_clean":"2024-09-11T05:09:22.202086+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:22.202086+0000","last_undegraded":"2024-09-11T05:09:22.202086+0000","last_fullsized":"2024-09-11T05:09:22.202086+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17154,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.199352+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:22.199352+0000","last_peered":"2024-09-11T05:09:22.199352+0000","last_clean":"2024-09-11T05:09:22.199352+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:22.199352+0000","last_undegraded":"2024-09-11T05:09:22.199352+0000","last_fullsized":"2024-09-11T05:09:22.199352+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":14791,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:25.403696+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:25.403696+0000","last_peered":"2024-09-11T05:09:25.403696+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:25.403696+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17496,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201993+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:22.201993+0000","last_peered":"2024-09-11T05:09:22.201993+0000","last_clean":"2024-09-11T05:09:22.201993+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:22.201993+0000","last_undegraded":"2024-09-11T05:09:22.201993+0000","last_fullsized":"2024-09-11T05:09:22.201993+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17719,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201739+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:22.201739+0000","last_peered":"2024-09-11T05:09:22.201739+0000","last_clean":"2024-09-11T05:09:22.201739+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:22.201739+0000","last_undegraded":"2024-09-11T05:09:22.201739+0000","last_fullsized":"2024-09-11T05:09:22.201739+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12945,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:22.198342+0000","last_peered":"2024-09-11T05:09:22.198342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:22.198342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17310,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.198190+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:22.198190+0000","last_peered":"2024-09-11T05:09:22.198190+0000","last_clean":"2024-09-11T05:09:22.198190+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:22.198190+0000","last_undegraded":"2024-09-11T05:09:22.198190+0000","last_fullsized":"2024-09-11T05:09:22.198190+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12805,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198381+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:22.198381+0000","last_peered":"2024-09-11T05:09:22.198381+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:22.198381+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17023,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202169+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:22.202169+0000","last_peered":"2024-09-11T05:09:22.202169+0000","last_clean":"2024-09-11T05:09:22.202169+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:22.202169+0000","last_undegraded":"2024-09-11T05:09:22.202169+0000","last_fullsized":"2024-09-11T05:09:22.202169+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17537,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202186+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:22.202186+0000","last_peered":"2024-09-11T05:09:22.202186+0000","last_clean":"2024-09-11T05:09:22.202186+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:22.202186+0000","last_undegraded":"2024-09-11T05:09:22.202186+0000","last_fullsized":"2024-09-11T05:09:22.202186+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12733,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201490+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:22.201490+0000","last_peered":"2024-09-11T05:09:22.201490+0000","last_clean":"2024-09-11T05:09:22.201490+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:22.201490+0000","last_undegraded":"2024-09-11T05:09:22.201490+0000","last_fullsized":"2024-09-11T05:09:22.201490+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12564,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.197864+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:22.197864+0000","last_peered":"2024-09-11T05:09:22.197864+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:22.197864+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12927,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201848+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:22.201848+0000","last_peered":"2024-09-11T05:09:22.201848+0000","last_clean":"2024-09-11T05:09:22.201848+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:22.201848+0000","last_undegraded":"2024-09-11T05:09:22.201848+0000","last_fullsized":"2024-09-11T05:09:22.201848+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744328,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627512,"statfs":{"total":96636764160,"available":95874572288,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721608,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650232,"statfs":{"total":96636764160,"available":95897837568,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631844,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739996,"statfs":{"total":96636764160,"available":95989755904,"internally_reserved":0,"allocated":479232,"data_stored":203603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:32.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:32.692 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:32.693 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:32.708 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-11T05:09:32.229037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811512,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675848,"statfs":{"total":386547056640,"available":383668068352,"internally_reserved":0,"allocated":1810432,"data_stored":707916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12159,"num_write_kb":-48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.032534"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12647,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198808+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:22.198808+0000","last_peered":"2024-09-11T05:09:22.198808+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:22.198808+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12899,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202048+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:22.202048+0000","last_peered":"2024-09-11T05:09:22.202048+0000","last_clean":"2024-09-11T05:09:22.202048+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:22.202048+0000","last_undegraded":"2024-09-11T05:09:22.202048+0000","last_fullsized":"2024-09-11T05:09:22.202048+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16745,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:23.176199+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:23.176199+0000","last_peered":"2024-09-11T05:09:23.176199+0000","last_clean":"2024-09-11T05:09:23.176199+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:23.176199+0000","last_undegraded":"2024-09-11T05:09:23.176199+0000","last_fullsized":"2024-09-11T05:09:23.176199+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13966,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202086+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:22.202086+0000","last_peered":"2024-09-11T05:09:22.202086+0000","last_clean":"2024-09-11T05:09:22.202086+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:22.202086+0000","last_undegraded":"2024-09-11T05:09:22.202086+0000","last_fullsized":"2024-09-11T05:09:22.202086+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17154,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.199352+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:22.199352+0000","last_peered":"2024-09-11T05:09:22.199352+0000","last_clean":"2024-09-11T05:09:22.199352+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:22.199352+0000","last_undegraded":"2024-09-11T05:09:22.199352+0000","last_fullsized":"2024-09-11T05:09:22.199352+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":16869,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:30.404962+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:30.404962+0000","last_peered":"2024-09-11T05:09:30.404962+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:30.404962+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17496,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201993+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:22.201993+0000","last_peered":"2024-09-11T05:09:22.201993+0000","last_clean":"2024-09-11T05:09:22.201993+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:22.201993+0000","last_undegraded":"2024-09-11T05:09:22.201993+0000","last_fullsized":"2024-09-11T05:09:22.201993+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17719,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201739+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:22.201739+0000","last_peered":"2024-09-11T05:09:22.201739+0000","last_clean":"2024-09-11T05:09:22.201739+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:22.201739+0000","last_undegraded":"2024-09-11T05:09:22.201739+0000","last_fullsized":"2024-09-11T05:09:22.201739+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12945,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:22.198342+0000","last_peered":"2024-09-11T05:09:22.198342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:22.198342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17310,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.198190+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:22.198190+0000","last_peered":"2024-09-11T05:09:22.198190+0000","last_clean":"2024-09-11T05:09:22.198190+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:22.198190+0000","last_undegraded":"2024-09-11T05:09:22.198190+0000","last_fullsized":"2024-09-11T05:09:22.198190+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12805,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198381+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:22.198381+0000","last_peered":"2024-09-11T05:09:22.198381+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:22.198381+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17023,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202169+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:22.202169+0000","last_peered":"2024-09-11T05:09:22.202169+0000","last_clean":"2024-09-11T05:09:22.202169+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:22.202169+0000","last_undegraded":"2024-09-11T05:09:22.202169+0000","last_fullsized":"2024-09-11T05:09:22.202169+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17537,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202186+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:22.202186+0000","last_peered":"2024-09-11T05:09:22.202186+0000","last_clean":"2024-09-11T05:09:22.202186+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:22.202186+0000","last_undegraded":"2024-09-11T05:09:22.202186+0000","last_fullsized":"2024-09-11T05:09:22.202186+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12733,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201490+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:22.201490+0000","last_peered":"2024-09-11T05:09:22.201490+0000","last_clean":"2024-09-11T05:09:22.201490+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:22.201490+0000","last_undegraded":"2024-09-11T05:09:22.201490+0000","last_fullsized":"2024-09-11T05:09:22.201490+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12564,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.197864+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:22.197864+0000","last_peered":"2024-09-11T05:09:22.197864+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:22.197864+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12927,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201848+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:22.201848+0000","last_peered":"2024-09-11T05:09:22.201848+0000","last_clean":"2024-09-11T05:09:22.201848+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:22.201848+0000","last_undegraded":"2024-09-11T05:09:22.201848+0000","last_fullsized":"2024-09-11T05:09:22.201848+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744328,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627512,"statfs":{"total":96636764160,"available":95874572288,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721608,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650232,"statfs":{"total":96636764160,"available":95897837568,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623516,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748324,"statfs":{"total":96636764160,"available":95998283776,"internally_reserved":0,"allocated":471040,"data_stored":195411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:32.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:33.034 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:33.034 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:33.047 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-11T05:09:32.229037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811512,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675848,"statfs":{"total":386547056640,"available":383668068352,"internally_reserved":0,"allocated":1810432,"data_stored":707916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12159,"num_write_kb":-48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.032534"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12647,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198808+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:22.198808+0000","last_peered":"2024-09-11T05:09:22.198808+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:22.198808+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12899,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202048+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:22.202048+0000","last_peered":"2024-09-11T05:09:22.202048+0000","last_clean":"2024-09-11T05:09:22.202048+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:22.202048+0000","last_undegraded":"2024-09-11T05:09:22.202048+0000","last_fullsized":"2024-09-11T05:09:22.202048+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16745,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:23.176199+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:23.176199+0000","last_peered":"2024-09-11T05:09:23.176199+0000","last_clean":"2024-09-11T05:09:23.176199+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:23.176199+0000","last_undegraded":"2024-09-11T05:09:23.176199+0000","last_fullsized":"2024-09-11T05:09:23.176199+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13966,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202086+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:22.202086+0000","last_peered":"2024-09-11T05:09:22.202086+0000","last_clean":"2024-09-11T05:09:22.202086+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:22.202086+0000","last_undegraded":"2024-09-11T05:09:22.202086+0000","last_fullsized":"2024-09-11T05:09:22.202086+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17154,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.199352+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:22.199352+0000","last_peered":"2024-09-11T05:09:22.199352+0000","last_clean":"2024-09-11T05:09:22.199352+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:22.199352+0000","last_undegraded":"2024-09-11T05:09:22.199352+0000","last_fullsized":"2024-09-11T05:09:22.199352+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":16869,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:30.404962+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:30.404962+0000","last_peered":"2024-09-11T05:09:30.404962+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:30.404962+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17496,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201993+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:22.201993+0000","last_peered":"2024-09-11T05:09:22.201993+0000","last_clean":"2024-09-11T05:09:22.201993+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:22.201993+0000","last_undegraded":"2024-09-11T05:09:22.201993+0000","last_fullsized":"2024-09-11T05:09:22.201993+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17719,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201739+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:22.201739+0000","last_peered":"2024-09-11T05:09:22.201739+0000","last_clean":"2024-09-11T05:09:22.201739+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:22.201739+0000","last_undegraded":"2024-09-11T05:09:22.201739+0000","last_fullsized":"2024-09-11T05:09:22.201739+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12945,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:22.198342+0000","last_peered":"2024-09-11T05:09:22.198342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:22.198342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17310,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.198190+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:22.198190+0000","last_peered":"2024-09-11T05:09:22.198190+0000","last_clean":"2024-09-11T05:09:22.198190+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:22.198190+0000","last_undegraded":"2024-09-11T05:09:22.198190+0000","last_fullsized":"2024-09-11T05:09:22.198190+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12805,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198381+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:22.198381+0000","last_peered":"2024-09-11T05:09:22.198381+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:22.198381+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17023,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202169+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:22.202169+0000","last_peered":"2024-09-11T05:09:22.202169+0000","last_clean":"2024-09-11T05:09:22.202169+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:22.202169+0000","last_undegraded":"2024-09-11T05:09:22.202169+0000","last_fullsized":"2024-09-11T05:09:22.202169+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17537,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202186+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:22.202186+0000","last_peered":"2024-09-11T05:09:22.202186+0000","last_clean":"2024-09-11T05:09:22.202186+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:22.202186+0000","last_undegraded":"2024-09-11T05:09:22.202186+0000","last_fullsized":"2024-09-11T05:09:22.202186+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12733,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201490+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:22.201490+0000","last_peered":"2024-09-11T05:09:22.201490+0000","last_clean":"2024-09-11T05:09:22.201490+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:22.201490+0000","last_undegraded":"2024-09-11T05:09:22.201490+0000","last_fullsized":"2024-09-11T05:09:22.201490+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12564,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.197864+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:22.197864+0000","last_peered":"2024-09-11T05:09:22.197864+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:22.197864+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12927,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201848+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:22.201848+0000","last_peered":"2024-09-11T05:09:22.201848+0000","last_clean":"2024-09-11T05:09:22.201848+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:22.201848+0000","last_undegraded":"2024-09-11T05:09:22.201848+0000","last_fullsized":"2024-09-11T05:09:22.201848+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744328,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627512,"statfs":{"total":96636764160,"available":95874572288,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721608,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650232,"statfs":{"total":96636764160,"available":95897837568,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623516,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748324,"statfs":{"total":96636764160,"available":95998283776,"internally_reserved":0,"allocated":471040,"data_stored":195411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722060,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649780,"statfs":{"total":96636764160,"available":95897374720,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:33.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:33.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12647, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:22.198808+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:22.198808+0000', 'last_peered': '2024-09-11T05:09:22.198808+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:22.198808+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:33.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-11T05:09:33.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12640", 'reported_seq': 16869, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:30.404962+0000', 'last_change': '2024-09-11T05:09:20.357434+0000', 'last_active': '2024-09-11T05:09:30.404962+0000', 'last_peered': '2024-09-11T05:09:30.404962+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.593525+0000', 'last_became_peered': '2024-09-11T04:56:20.593525+0000', 'last_unstale': '2024-09-11T05:09:30.404962+0000', 'last_undegraded': '2024-09-11T04:56:20.257598+0000', 'last_fullsized': '2024-09-11T04:56:20.228049+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12640, 'num_write_kb': 50560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2069, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -252}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:33.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:33.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12945, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:22.198342+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:09:22.198342+0000', 'last_peered': '2024-09-11T05:09:22.198342+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:22.198342+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:33.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:33.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12805, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:22.198381+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:22.198381+0000', 'last_peered': '2024-09-11T05:09:22.198381+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:22.198381+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:33.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:33.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 12564, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:22.197864+0000', 'last_change': '2024-09-11T04:56:20.668083+0000', 'last_active': '2024-09-11T05:09:22.197864+0000', 'last_peered': '2024-09-11T05:09:22.197864+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:22.197864+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:33.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:33.526 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:33.526 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9518/36 objects degraded (26438.889%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":900,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2887516160,"bytes_avail":383659540480,"bytes_total":386547056640,"degraded_objects":9518,"degraded_total":36,"degraded_ratio":264.38888888888891},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:33.543 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:09:33.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:33.894 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:33.894 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:33.907 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-11T05:09:33.231931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811516,"kb_used_data":1776,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675844,"statfs":{"total":386547056640,"available":383668064256,"internally_reserved":0,"allocated":1818624,"data_stored":711132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6736,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12159,"num_write_kb":-48631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.285382"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12647,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198808+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:22.198808+0000","last_peered":"2024-09-11T05:09:22.198808+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:22.198808+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12899,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202048+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:22.202048+0000","last_peered":"2024-09-11T05:09:22.202048+0000","last_clean":"2024-09-11T05:09:22.202048+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:22.202048+0000","last_undegraded":"2024-09-11T05:09:22.202048+0000","last_fullsized":"2024-09-11T05:09:22.202048+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16745,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:23.176199+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:23.176199+0000","last_peered":"2024-09-11T05:09:23.176199+0000","last_clean":"2024-09-11T05:09:23.176199+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:23.176199+0000","last_undegraded":"2024-09-11T05:09:23.176199+0000","last_fullsized":"2024-09-11T05:09:23.176199+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13966,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202086+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:22.202086+0000","last_peered":"2024-09-11T05:09:22.202086+0000","last_clean":"2024-09-11T05:09:22.202086+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:22.202086+0000","last_undegraded":"2024-09-11T05:09:22.202086+0000","last_fullsized":"2024-09-11T05:09:22.202086+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17154,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.199352+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:22.199352+0000","last_peered":"2024-09-11T05:09:22.199352+0000","last_clean":"2024-09-11T05:09:22.199352+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:22.199352+0000","last_undegraded":"2024-09-11T05:09:22.199352+0000","last_fullsized":"2024-09-11T05:09:22.199352+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":16869,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:30.404962+0000","last_change":"2024-09-11T05:09:20.357434+0000","last_active":"2024-09-11T05:09:30.404962+0000","last_peered":"2024-09-11T05:09:30.404962+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.593525+0000","last_became_peered":"2024-09-11T04:56:20.593525+0000","last_unstale":"2024-09-11T05:09:30.404962+0000","last_undegraded":"2024-09-11T04:56:20.257598+0000","last_fullsized":"2024-09-11T04:56:20.228049+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17498,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-11T05:09:32.237396+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:32.237396+0000","last_peered":"2024-09-11T05:09:32.237396+0000","last_clean":"2024-09-11T05:09:32.237396+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:32.237396+0000","last_undegraded":"2024-09-11T05:09:32.237396+0000","last_fullsized":"2024-09-11T05:09:32.237396+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17719,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201739+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:22.201739+0000","last_peered":"2024-09-11T05:09:22.201739+0000","last_clean":"2024-09-11T05:09:22.201739+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:22.201739+0000","last_undegraded":"2024-09-11T05:09:22.201739+0000","last_fullsized":"2024-09-11T05:09:22.201739+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12945,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198342+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:22.198342+0000","last_peered":"2024-09-11T05:09:22.198342+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:22.198342+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17310,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.198190+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:22.198190+0000","last_peered":"2024-09-11T05:09:22.198190+0000","last_clean":"2024-09-11T05:09:22.198190+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:22.198190+0000","last_undegraded":"2024-09-11T05:09:22.198190+0000","last_fullsized":"2024-09-11T05:09:22.198190+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12805,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.198381+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:22.198381+0000","last_peered":"2024-09-11T05:09:22.198381+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:22.198381+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17023,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202169+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:22.202169+0000","last_peered":"2024-09-11T05:09:22.202169+0000","last_clean":"2024-09-11T05:09:22.202169+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:22.202169+0000","last_undegraded":"2024-09-11T05:09:22.202169+0000","last_fullsized":"2024-09-11T05:09:22.202169+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17537,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202186+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:22.202186+0000","last_peered":"2024-09-11T05:09:22.202186+0000","last_clean":"2024-09-11T05:09:22.202186+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:22.202186+0000","last_undegraded":"2024-09-11T05:09:22.202186+0000","last_fullsized":"2024-09-11T05:09:22.202186+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12733,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201490+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:22.201490+0000","last_peered":"2024-09-11T05:09:22.201490+0000","last_clean":"2024-09-11T05:09:22.201490+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:22.201490+0000","last_undegraded":"2024-09-11T05:09:22.201490+0000","last_fullsized":"2024-09-11T05:09:22.201490+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12564,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:22.197864+0000","last_change":"2024-09-11T04:56:20.668083+0000","last_active":"2024-09-11T05:09:22.197864+0000","last_peered":"2024-09-11T05:09:22.197864+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:22.197864+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12927,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201848+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:22.201848+0000","last_peered":"2024-09-11T05:09:22.201848+0000","last_clean":"2024-09-11T05:09:22.201848+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:22.201848+0000","last_undegraded":"2024-09-11T05:09:22.201848+0000","last_fullsized":"2024-09-11T05:09:22.201848+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744328,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627512,"statfs":{"total":96636764160,"available":95874572288,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721608,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650232,"statfs":{"total":96636764160,"available":95897837568,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623516,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748324,"statfs":{"total":96636764160,"available":95998283776,"internally_reserved":0,"allocated":471040,"data_stored":195411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":454656,"data_stored":174051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:36.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:37.257 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:37.257 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:37.270 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-11T05:09:35.948547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811556,"kb_used_data":1808,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675804,"statfs":{"total":386547056640,"available":383668023296,"internally_reserved":0,"allocated":1851392,"data_stored":723954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001349"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12651,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.233381+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:33.233381+0000","last_peered":"2024-09-11T05:09:33.233381+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:33.233381+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12899,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202048+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:22.202048+0000","last_peered":"2024-09-11T05:09:22.202048+0000","last_clean":"2024-09-11T05:09:22.202048+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:22.202048+0000","last_undegraded":"2024-09-11T05:09:22.202048+0000","last_fullsized":"2024-09-11T05:09:22.202048+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16749,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.235024+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:33.235024+0000","last_peered":"2024-09-11T05:09:33.235024+0000","last_clean":"2024-09-11T05:09:33.235024+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:33.235024+0000","last_undegraded":"2024-09-11T05:09:33.235024+0000","last_fullsized":"2024-09-11T05:09:33.235024+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13966,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202086+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:22.202086+0000","last_peered":"2024-09-11T05:09:22.202086+0000","last_clean":"2024-09-11T05:09:22.202086+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:22.202086+0000","last_undegraded":"2024-09-11T05:09:22.202086+0000","last_fullsized":"2024-09-11T05:09:22.202086+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17158,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.232477+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:33.232477+0000","last_peered":"2024-09-11T05:09:33.232477+0000","last_clean":"2024-09-11T05:09:33.232477+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:33.232477+0000","last_undegraded":"2024-09-11T05:09:33.232477+0000","last_fullsized":"2024-09-11T05:09:33.232477+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17386,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.322723+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:33.322723+0000","last_peered":"2024-09-11T05:09:33.322723+0000","last_clean":"2024-09-11T05:09:33.322723+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:33.322723+0000","last_undegraded":"2024-09-11T05:09:33.322723+0000","last_fullsized":"2024-09-11T05:09:33.322723+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17498,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-11T05:09:32.237396+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:32.237396+0000","last_peered":"2024-09-11T05:09:32.237396+0000","last_clean":"2024-09-11T05:09:32.237396+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:32.237396+0000","last_undegraded":"2024-09-11T05:09:32.237396+0000","last_fullsized":"2024-09-11T05:09:32.237396+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17719,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201739+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:22.201739+0000","last_peered":"2024-09-11T05:09:22.201739+0000","last_clean":"2024-09-11T05:09:22.201739+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:22.201739+0000","last_undegraded":"2024-09-11T05:09:22.201739+0000","last_fullsized":"2024-09-11T05:09:22.201739+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12949,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.232817+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:33.232817+0000","last_peered":"2024-09-11T05:09:33.232817+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:33.232817+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17314,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.234896+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:33.234896+0000","last_peered":"2024-09-11T05:09:33.234896+0000","last_clean":"2024-09-11T05:09:33.234896+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:33.234896+0000","last_undegraded":"2024-09-11T05:09:33.234896+0000","last_fullsized":"2024-09-11T05:09:33.234896+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12809,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.235247+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:33.235247+0000","last_peered":"2024-09-11T05:09:33.235247+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:33.235247+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17023,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202169+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:22.202169+0000","last_peered":"2024-09-11T05:09:22.202169+0000","last_clean":"2024-09-11T05:09:22.202169+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:22.202169+0000","last_undegraded":"2024-09-11T05:09:22.202169+0000","last_fullsized":"2024-09-11T05:09:22.202169+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17537,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202186+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:22.202186+0000","last_peered":"2024-09-11T05:09:22.202186+0000","last_clean":"2024-09-11T05:09:22.202186+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:22.202186+0000","last_undegraded":"2024-09-11T05:09:22.202186+0000","last_fullsized":"2024-09-11T05:09:22.202186+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12733,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201490+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:22.201490+0000","last_peered":"2024-09-11T05:09:22.201490+0000","last_clean":"2024-09-11T05:09:22.201490+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:22.201490+0000","last_undegraded":"2024-09-11T05:09:22.201490+0000","last_fullsized":"2024-09-11T05:09:22.201490+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":13869,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:35.074673+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:35.074673+0000","last_peered":"2024-09-11T05:09:35.074673+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:35.074673+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1653}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12927,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201848+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:22.201848+0000","last_peered":"2024-09-11T05:09:22.201848+0000","last_clean":"2024-09-11T05:09:22.201848+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:22.201848+0000","last_undegraded":"2024-09-11T05:09:22.201848+0000","last_fullsized":"2024-09-11T05:09:22.201848+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744348,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627492,"statfs":{"total":96636764160,"available":95874551808,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623516,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748324,"statfs":{"total":96636764160,"available":95998283776,"internally_reserved":0,"allocated":471040,"data_stored":195411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":454656,"data_stored":174051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:37.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:37.582 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:37.583 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:37.596 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-11T05:09:35.948547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811556,"kb_used_data":1808,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675804,"statfs":{"total":386547056640,"available":383668023296,"internally_reserved":0,"allocated":1851392,"data_stored":723954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001349"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12651,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.233381+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:33.233381+0000","last_peered":"2024-09-11T05:09:33.233381+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:33.233381+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12899,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202048+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:22.202048+0000","last_peered":"2024-09-11T05:09:22.202048+0000","last_clean":"2024-09-11T05:09:22.202048+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:22.202048+0000","last_undegraded":"2024-09-11T05:09:22.202048+0000","last_fullsized":"2024-09-11T05:09:22.202048+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16749,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.235024+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:33.235024+0000","last_peered":"2024-09-11T05:09:33.235024+0000","last_clean":"2024-09-11T05:09:33.235024+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:33.235024+0000","last_undegraded":"2024-09-11T05:09:33.235024+0000","last_fullsized":"2024-09-11T05:09:33.235024+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13966,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202086+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:22.202086+0000","last_peered":"2024-09-11T05:09:22.202086+0000","last_clean":"2024-09-11T05:09:22.202086+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:22.202086+0000","last_undegraded":"2024-09-11T05:09:22.202086+0000","last_fullsized":"2024-09-11T05:09:22.202086+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17158,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.232477+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:33.232477+0000","last_peered":"2024-09-11T05:09:33.232477+0000","last_clean":"2024-09-11T05:09:33.232477+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:33.232477+0000","last_undegraded":"2024-09-11T05:09:33.232477+0000","last_fullsized":"2024-09-11T05:09:33.232477+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17386,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.322723+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:33.322723+0000","last_peered":"2024-09-11T05:09:33.322723+0000","last_clean":"2024-09-11T05:09:33.322723+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:33.322723+0000","last_undegraded":"2024-09-11T05:09:33.322723+0000","last_fullsized":"2024-09-11T05:09:33.322723+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17498,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-11T05:09:32.237396+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:32.237396+0000","last_peered":"2024-09-11T05:09:32.237396+0000","last_clean":"2024-09-11T05:09:32.237396+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:32.237396+0000","last_undegraded":"2024-09-11T05:09:32.237396+0000","last_fullsized":"2024-09-11T05:09:32.237396+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17719,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201739+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:22.201739+0000","last_peered":"2024-09-11T05:09:22.201739+0000","last_clean":"2024-09-11T05:09:22.201739+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:22.201739+0000","last_undegraded":"2024-09-11T05:09:22.201739+0000","last_fullsized":"2024-09-11T05:09:22.201739+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12949,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.232817+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:33.232817+0000","last_peered":"2024-09-11T05:09:33.232817+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:33.232817+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17314,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.234896+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:33.234896+0000","last_peered":"2024-09-11T05:09:33.234896+0000","last_clean":"2024-09-11T05:09:33.234896+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:33.234896+0000","last_undegraded":"2024-09-11T05:09:33.234896+0000","last_fullsized":"2024-09-11T05:09:33.234896+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12809,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.235247+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:33.235247+0000","last_peered":"2024-09-11T05:09:33.235247+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:33.235247+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17023,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202169+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:22.202169+0000","last_peered":"2024-09-11T05:09:22.202169+0000","last_clean":"2024-09-11T05:09:22.202169+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:22.202169+0000","last_undegraded":"2024-09-11T05:09:22.202169+0000","last_fullsized":"2024-09-11T05:09:22.202169+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17537,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.202186+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:22.202186+0000","last_peered":"2024-09-11T05:09:22.202186+0000","last_clean":"2024-09-11T05:09:22.202186+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:22.202186+0000","last_undegraded":"2024-09-11T05:09:22.202186+0000","last_fullsized":"2024-09-11T05:09:22.202186+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12733,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201490+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:22.201490+0000","last_peered":"2024-09-11T05:09:22.201490+0000","last_clean":"2024-09-11T05:09:22.201490+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:22.201490+0000","last_undegraded":"2024-09-11T05:09:22.201490+0000","last_fullsized":"2024-09-11T05:09:22.201490+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":13869,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:35.074673+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:35.074673+0000","last_peered":"2024-09-11T05:09:35.074673+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:35.074673+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1653}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12927,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-11T05:09:22.201848+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:22.201848+0000","last_peered":"2024-09-11T05:09:22.201848+0000","last_clean":"2024-09-11T05:09:22.201848+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:22.201848+0000","last_undegraded":"2024-09-11T05:09:22.201848+0000","last_fullsized":"2024-09-11T05:09:22.201848+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744348,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627492,"statfs":{"total":96636764160,"available":95874551808,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623516,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748324,"statfs":{"total":96636764160,"available":95998283776,"internally_reserved":0,"allocated":471040,"data_stored":195411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":454656,"data_stored":174051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:37.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:37.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12651, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:33.233381+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:33.233381+0000', 'last_peered': '2024-09-11T05:09:33.233381+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:33.233381+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:37.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:37.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12949, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:33.232817+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:09:33.232817+0000', 'last_peered': '2024-09-11T05:09:33.232817+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:33.232817+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:37.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:37.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12809, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:33.235247+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:33.235247+0000', 'last_peered': '2024-09-11T05:09:33.235247+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:33.235247+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:37.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:37.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 13869, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:35.074673+0000', 'last_change': '2024-09-11T05:09:31.310757+0000', 'last_active': '2024-09-11T05:09:35.074673+0000', 'last_peered': '2024-09-11T05:09:35.074673+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:35.074673+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1655, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 667, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1653}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:37.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:38.000 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:38.000 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8616/36 objects degraded (23933.333%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":905,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2879033344,"bytes_avail":383668023296,"bytes_total":386547056640,"degraded_objects":8616,"degraded_total":36,"degraded_ratio":239.33333333333337,"recovering_objects_per_sec":242,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1941,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:38.013 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:09:38.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:38.328 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:38.340 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-09-11T05:09:37.948858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819904,"kb_used_data":1832,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667456,"statfs":{"total":386547056640,"available":383659474944,"internally_reserved":0,"allocated":1875968,"data_stored":733560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001283"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12651,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.233381+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:33.233381+0000","last_peered":"2024-09-11T05:09:33.233381+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:33.233381+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12903,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238269+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:33.238269+0000","last_peered":"2024-09-11T05:09:33.238269+0000","last_clean":"2024-09-11T05:09:33.238269+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:33.238269+0000","last_undegraded":"2024-09-11T05:09:33.238269+0000","last_fullsized":"2024-09-11T05:09:33.238269+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16749,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.235024+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:33.235024+0000","last_peered":"2024-09-11T05:09:33.235024+0000","last_clean":"2024-09-11T05:09:33.235024+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:33.235024+0000","last_undegraded":"2024-09-11T05:09:33.235024+0000","last_fullsized":"2024-09-11T05:09:33.235024+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13970,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237645+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:33.237645+0000","last_peered":"2024-09-11T05:09:33.237645+0000","last_clean":"2024-09-11T05:09:33.237645+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:33.237645+0000","last_undegraded":"2024-09-11T05:09:33.237645+0000","last_fullsized":"2024-09-11T05:09:33.237645+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17158,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.232477+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:33.232477+0000","last_peered":"2024-09-11T05:09:33.232477+0000","last_clean":"2024-09-11T05:09:33.232477+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:33.232477+0000","last_undegraded":"2024-09-11T05:09:33.232477+0000","last_fullsized":"2024-09-11T05:09:33.232477+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17386,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.322723+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:33.322723+0000","last_peered":"2024-09-11T05:09:33.322723+0000","last_clean":"2024-09-11T05:09:33.322723+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:33.322723+0000","last_undegraded":"2024-09-11T05:09:33.322723+0000","last_fullsized":"2024-09-11T05:09:33.322723+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.236494+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:33.236494+0000","last_peered":"2024-09-11T05:09:33.236494+0000","last_clean":"2024-09-11T05:09:33.236494+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:33.236494+0000","last_undegraded":"2024-09-11T05:09:33.236494+0000","last_fullsized":"2024-09-11T05:09:33.236494+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17723,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237530+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:33.237530+0000","last_peered":"2024-09-11T05:09:33.237530+0000","last_clean":"2024-09-11T05:09:33.237530+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:33.237530+0000","last_undegraded":"2024-09-11T05:09:33.237530+0000","last_fullsized":"2024-09-11T05:09:33.237530+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12949,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.232817+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:33.232817+0000","last_peered":"2024-09-11T05:09:33.232817+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:33.232817+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17314,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.234896+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:33.234896+0000","last_peered":"2024-09-11T05:09:33.234896+0000","last_clean":"2024-09-11T05:09:33.234896+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:33.234896+0000","last_undegraded":"2024-09-11T05:09:33.234896+0000","last_fullsized":"2024-09-11T05:09:33.234896+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12809,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.235247+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:33.235247+0000","last_peered":"2024-09-11T05:09:33.235247+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:33.235247+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17027,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238337+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:33.238337+0000","last_peered":"2024-09-11T05:09:33.238337+0000","last_clean":"2024-09-11T05:09:33.238337+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:33.238337+0000","last_undegraded":"2024-09-11T05:09:33.238337+0000","last_fullsized":"2024-09-11T05:09:33.238337+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17541,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237785+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:33.237785+0000","last_peered":"2024-09-11T05:09:33.237785+0000","last_clean":"2024-09-11T05:09:33.237785+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:33.237785+0000","last_undegraded":"2024-09-11T05:09:33.237785+0000","last_fullsized":"2024-09-11T05:09:33.237785+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12737,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237828+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:33.237828+0000","last_peered":"2024-09-11T05:09:33.237828+0000","last_clean":"2024-09-11T05:09:33.237828+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:33.237828+0000","last_undegraded":"2024-09-11T05:09:33.237828+0000","last_fullsized":"2024-09-11T05:09:33.237828+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":13869,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:35.074673+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:35.074673+0000","last_peered":"2024-09-11T05:09:35.074673+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:35.074673+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1653}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12931,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237643+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:33.237643+0000","last_peered":"2024-09-11T05:09:33.237643+0000","last_clean":"2024-09-11T05:09:33.237643+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:33.237643+0000","last_undegraded":"2024-09-11T05:09:33.237643+0000","last_fullsized":"2024-09-11T05:09:33.237643+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744348,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627492,"statfs":{"total":96636764160,"available":95874551808,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631852,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739988,"statfs":{"total":96636764160,"available":95989747712,"internally_reserved":0,"allocated":487424,"data_stored":201822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:41.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:41.676 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:41.689 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-11T05:09:39.949165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819904,"kb_used_data":1832,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667456,"statfs":{"total":386547056640,"available":383659474944,"internally_reserved":0,"allocated":1875968,"data_stored":733560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001279"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12651,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.233381+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:33.233381+0000","last_peered":"2024-09-11T05:09:33.233381+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:33.233381+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12903,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238269+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:33.238269+0000","last_peered":"2024-09-11T05:09:33.238269+0000","last_clean":"2024-09-11T05:09:33.238269+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:33.238269+0000","last_undegraded":"2024-09-11T05:09:33.238269+0000","last_fullsized":"2024-09-11T05:09:33.238269+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16749,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.235024+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:33.235024+0000","last_peered":"2024-09-11T05:09:33.235024+0000","last_clean":"2024-09-11T05:09:33.235024+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:33.235024+0000","last_undegraded":"2024-09-11T05:09:33.235024+0000","last_fullsized":"2024-09-11T05:09:33.235024+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13970,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237645+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:33.237645+0000","last_peered":"2024-09-11T05:09:33.237645+0000","last_clean":"2024-09-11T05:09:33.237645+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:33.237645+0000","last_undegraded":"2024-09-11T05:09:33.237645+0000","last_fullsized":"2024-09-11T05:09:33.237645+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17158,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.232477+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:33.232477+0000","last_peered":"2024-09-11T05:09:33.232477+0000","last_clean":"2024-09-11T05:09:33.232477+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:33.232477+0000","last_undegraded":"2024-09-11T05:09:33.232477+0000","last_fullsized":"2024-09-11T05:09:33.232477+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17386,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.322723+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:33.322723+0000","last_peered":"2024-09-11T05:09:33.322723+0000","last_clean":"2024-09-11T05:09:33.322723+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:33.322723+0000","last_undegraded":"2024-09-11T05:09:33.322723+0000","last_fullsized":"2024-09-11T05:09:33.322723+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.236494+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:33.236494+0000","last_peered":"2024-09-11T05:09:33.236494+0000","last_clean":"2024-09-11T05:09:33.236494+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:33.236494+0000","last_undegraded":"2024-09-11T05:09:33.236494+0000","last_fullsized":"2024-09-11T05:09:33.236494+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17723,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237530+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:33.237530+0000","last_peered":"2024-09-11T05:09:33.237530+0000","last_clean":"2024-09-11T05:09:33.237530+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:33.237530+0000","last_undegraded":"2024-09-11T05:09:33.237530+0000","last_fullsized":"2024-09-11T05:09:33.237530+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12949,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.232817+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:33.232817+0000","last_peered":"2024-09-11T05:09:33.232817+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:33.232817+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17314,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.234896+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:33.234896+0000","last_peered":"2024-09-11T05:09:33.234896+0000","last_clean":"2024-09-11T05:09:33.234896+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:33.234896+0000","last_undegraded":"2024-09-11T05:09:33.234896+0000","last_fullsized":"2024-09-11T05:09:33.234896+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12809,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.235247+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:33.235247+0000","last_peered":"2024-09-11T05:09:33.235247+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:33.235247+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17027,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238337+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:33.238337+0000","last_peered":"2024-09-11T05:09:33.238337+0000","last_clean":"2024-09-11T05:09:33.238337+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:33.238337+0000","last_undegraded":"2024-09-11T05:09:33.238337+0000","last_fullsized":"2024-09-11T05:09:33.238337+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17541,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237785+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:33.237785+0000","last_peered":"2024-09-11T05:09:33.237785+0000","last_clean":"2024-09-11T05:09:33.237785+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:33.237785+0000","last_undegraded":"2024-09-11T05:09:33.237785+0000","last_fullsized":"2024-09-11T05:09:33.237785+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12737,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237828+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:33.237828+0000","last_peered":"2024-09-11T05:09:33.237828+0000","last_clean":"2024-09-11T05:09:33.237828+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:33.237828+0000","last_undegraded":"2024-09-11T05:09:33.237828+0000","last_fullsized":"2024-09-11T05:09:33.237828+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":13869,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:35.074673+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:35.074673+0000","last_peered":"2024-09-11T05:09:35.074673+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:35.074673+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1653}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12931,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237643+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:33.237643+0000","last_peered":"2024-09-11T05:09:33.237643+0000","last_clean":"2024-09-11T05:09:33.237643+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:33.237643+0000","last_undegraded":"2024-09-11T05:09:33.237643+0000","last_fullsized":"2024-09-11T05:09:33.237643+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744348,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627492,"statfs":{"total":96636764160,"available":95874551808,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631852,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739988,"statfs":{"total":96636764160,"available":95989747712,"internally_reserved":0,"allocated":487424,"data_stored":201822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:41.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:42.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T05:09:42.000+0000 7f613151d640 0 --2- 172.21.15.31:0/157736714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f612c073500 0x7f612c1262b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T05:09:42.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:42.222 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-11T05:09:41.949430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819904,"kb_used_data":1832,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667456,"statfs":{"total":386547056640,"available":383659474944,"internally_reserved":0,"allocated":1875968,"data_stored":733560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1819,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.720392"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12651,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.233381+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:33.233381+0000","last_peered":"2024-09-11T05:09:33.233381+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:33.233381+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12903,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238269+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:33.238269+0000","last_peered":"2024-09-11T05:09:33.238269+0000","last_clean":"2024-09-11T05:09:33.238269+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:33.238269+0000","last_undegraded":"2024-09-11T05:09:33.238269+0000","last_fullsized":"2024-09-11T05:09:33.238269+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16749,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.235024+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:33.235024+0000","last_peered":"2024-09-11T05:09:33.235024+0000","last_clean":"2024-09-11T05:09:33.235024+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:33.235024+0000","last_undegraded":"2024-09-11T05:09:33.235024+0000","last_fullsized":"2024-09-11T05:09:33.235024+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13970,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237645+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:33.237645+0000","last_peered":"2024-09-11T05:09:33.237645+0000","last_clean":"2024-09-11T05:09:33.237645+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:33.237645+0000","last_undegraded":"2024-09-11T05:09:33.237645+0000","last_fullsized":"2024-09-11T05:09:33.237645+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17158,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.232477+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:33.232477+0000","last_peered":"2024-09-11T05:09:33.232477+0000","last_clean":"2024-09-11T05:09:33.232477+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:33.232477+0000","last_undegraded":"2024-09-11T05:09:33.232477+0000","last_fullsized":"2024-09-11T05:09:33.232477+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17386,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.322723+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:33.322723+0000","last_peered":"2024-09-11T05:09:33.322723+0000","last_clean":"2024-09-11T05:09:33.322723+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:33.322723+0000","last_undegraded":"2024-09-11T05:09:33.322723+0000","last_fullsized":"2024-09-11T05:09:33.322723+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.236494+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:33.236494+0000","last_peered":"2024-09-11T05:09:33.236494+0000","last_clean":"2024-09-11T05:09:33.236494+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:33.236494+0000","last_undegraded":"2024-09-11T05:09:33.236494+0000","last_fullsized":"2024-09-11T05:09:33.236494+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17723,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237530+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:33.237530+0000","last_peered":"2024-09-11T05:09:33.237530+0000","last_clean":"2024-09-11T05:09:33.237530+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:33.237530+0000","last_undegraded":"2024-09-11T05:09:33.237530+0000","last_fullsized":"2024-09-11T05:09:33.237530+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12949,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.232817+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:33.232817+0000","last_peered":"2024-09-11T05:09:33.232817+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:33.232817+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17315,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.234947+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:33.234947+0000","last_peered":"2024-09-11T05:09:33.234947+0000","last_clean":"2024-09-11T05:09:33.234947+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:33.234947+0000","last_undegraded":"2024-09-11T05:09:33.234947+0000","last_fullsized":"2024-09-11T05:09:33.234947+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12809,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.235247+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:33.235247+0000","last_peered":"2024-09-11T05:09:33.235247+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:33.235247+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17027,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238337+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:33.238337+0000","last_peered":"2024-09-11T05:09:33.238337+0000","last_clean":"2024-09-11T05:09:33.238337+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:33.238337+0000","last_undegraded":"2024-09-11T05:09:33.238337+0000","last_fullsized":"2024-09-11T05:09:33.238337+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17541,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237785+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:33.237785+0000","last_peered":"2024-09-11T05:09:33.237785+0000","last_clean":"2024-09-11T05:09:33.237785+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:33.237785+0000","last_undegraded":"2024-09-11T05:09:33.237785+0000","last_fullsized":"2024-09-11T05:09:33.237785+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12737,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237828+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:33.237828+0000","last_peered":"2024-09-11T05:09:33.237828+0000","last_clean":"2024-09-11T05:09:33.237828+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:33.237828+0000","last_undegraded":"2024-09-11T05:09:33.237828+0000","last_fullsized":"2024-09-11T05:09:33.237828+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":15703,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:40.075790+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:40.075790+0000","last_peered":"2024-09-11T05:09:40.075790+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:40.075790+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-736}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12931,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237643+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:33.237643+0000","last_peered":"2024-09-11T05:09:33.237643+0000","last_clean":"2024-09-11T05:09:33.237643+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:33.237643+0000","last_undegraded":"2024-09-11T05:09:33.237643+0000","last_fullsized":"2024-09-11T05:09:33.237643+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744348,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627492,"statfs":{"total":96636764160,"available":95874551808,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631852,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739988,"statfs":{"total":96636764160,"available":95989747712,"internally_reserved":0,"allocated":487424,"data_stored":201822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:42.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:42.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12651, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:33.233381+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:33.233381+0000', 'last_peered': '2024-09-11T05:09:33.233381+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:33.233381+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:42.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:42.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 12949, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:33.232817+0000', 'last_change': '2024-09-11T04:56:20.627319+0000', 'last_active': '2024-09-11T05:09:33.232817+0000', 'last_peered': '2024-09-11T05:09:33.232817+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:33.232817+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:42.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:42.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12809, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:33.235247+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:33.235247+0000', 'last_peered': '2024-09-11T05:09:33.235247+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:33.235247+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:42.224 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:42.224 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 15703, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:40.075790+0000', 'last_change': '2024-09-11T05:09:31.310757+0000', 'last_active': '2024-09-11T05:09:40.075790+0000', 'last_peered': '2024-09-11T05:09:40.075790+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:40.075790+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 738, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1584, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -736}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:42.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:42.614 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:42.615 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7699/36 objects degraded (21386.111%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":909,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2887581696,"bytes_avail":383659474944,"bytes_total":386547056640,"degraded_objects":7699,"degraded_total":36,"degraded_ratio":213.86111111111114,"recovering_objects_per_sec":187,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1819,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:42.630 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:09:42.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:42.943 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:42.955 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-11T05:09:41.949430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819904,"kb_used_data":1832,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667456,"statfs":{"total":386547056640,"available":383659474944,"internally_reserved":0,"allocated":1875968,"data_stored":733560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1819,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.720392"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12651,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.233381+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:33.233381+0000","last_peered":"2024-09-11T05:09:33.233381+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:33.233381+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12903,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238269+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:33.238269+0000","last_peered":"2024-09-11T05:09:33.238269+0000","last_clean":"2024-09-11T05:09:33.238269+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:33.238269+0000","last_undegraded":"2024-09-11T05:09:33.238269+0000","last_fullsized":"2024-09-11T05:09:33.238269+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16749,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.235024+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:33.235024+0000","last_peered":"2024-09-11T05:09:33.235024+0000","last_clean":"2024-09-11T05:09:33.235024+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:33.235024+0000","last_undegraded":"2024-09-11T05:09:33.235024+0000","last_fullsized":"2024-09-11T05:09:33.235024+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13970,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237645+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:33.237645+0000","last_peered":"2024-09-11T05:09:33.237645+0000","last_clean":"2024-09-11T05:09:33.237645+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:33.237645+0000","last_undegraded":"2024-09-11T05:09:33.237645+0000","last_fullsized":"2024-09-11T05:09:33.237645+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17158,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.232477+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:33.232477+0000","last_peered":"2024-09-11T05:09:33.232477+0000","last_clean":"2024-09-11T05:09:33.232477+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:33.232477+0000","last_undegraded":"2024-09-11T05:09:33.232477+0000","last_fullsized":"2024-09-11T05:09:33.232477+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17386,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.322723+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:33.322723+0000","last_peered":"2024-09-11T05:09:33.322723+0000","last_clean":"2024-09-11T05:09:33.322723+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:33.322723+0000","last_undegraded":"2024-09-11T05:09:33.322723+0000","last_fullsized":"2024-09-11T05:09:33.322723+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.236494+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:33.236494+0000","last_peered":"2024-09-11T05:09:33.236494+0000","last_clean":"2024-09-11T05:09:33.236494+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:33.236494+0000","last_undegraded":"2024-09-11T05:09:33.236494+0000","last_fullsized":"2024-09-11T05:09:33.236494+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17723,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237530+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:33.237530+0000","last_peered":"2024-09-11T05:09:33.237530+0000","last_clean":"2024-09-11T05:09:33.237530+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:33.237530+0000","last_undegraded":"2024-09-11T05:09:33.237530+0000","last_fullsized":"2024-09-11T05:09:33.237530+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":12949,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.232817+0000","last_change":"2024-09-11T04:56:20.627319+0000","last_active":"2024-09-11T05:09:33.232817+0000","last_peered":"2024-09-11T05:09:33.232817+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:33.232817+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17315,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.234947+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:33.234947+0000","last_peered":"2024-09-11T05:09:33.234947+0000","last_clean":"2024-09-11T05:09:33.234947+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:33.234947+0000","last_undegraded":"2024-09-11T05:09:33.234947+0000","last_fullsized":"2024-09-11T05:09:33.234947+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12809,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:33.235247+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:33.235247+0000","last_peered":"2024-09-11T05:09:33.235247+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:33.235247+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17027,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238337+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:33.238337+0000","last_peered":"2024-09-11T05:09:33.238337+0000","last_clean":"2024-09-11T05:09:33.238337+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:33.238337+0000","last_undegraded":"2024-09-11T05:09:33.238337+0000","last_fullsized":"2024-09-11T05:09:33.238337+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17541,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237785+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:33.237785+0000","last_peered":"2024-09-11T05:09:33.237785+0000","last_clean":"2024-09-11T05:09:33.237785+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:33.237785+0000","last_undegraded":"2024-09-11T05:09:33.237785+0000","last_fullsized":"2024-09-11T05:09:33.237785+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12737,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237828+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:33.237828+0000","last_peered":"2024-09-11T05:09:33.237828+0000","last_clean":"2024-09-11T05:09:33.237828+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:33.237828+0000","last_undegraded":"2024-09-11T05:09:33.237828+0000","last_fullsized":"2024-09-11T05:09:33.237828+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":15703,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:40.075790+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:40.075790+0000","last_peered":"2024-09-11T05:09:40.075790+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:40.075790+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-736}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12931,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237643+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:33.237643+0000","last_peered":"2024-09-11T05:09:33.237643+0000","last_clean":"2024-09-11T05:09:33.237643+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:33.237643+0000","last_undegraded":"2024-09-11T05:09:33.237643+0000","last_fullsized":"2024-09-11T05:09:33.237643+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744348,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627492,"statfs":{"total":96636764160,"available":95874551808,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631852,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739988,"statfs":{"total":96636764160,"available":95989747712,"internally_reserved":0,"allocated":487424,"data_stored":201822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:45.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:46.276 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:46.277 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:46.289 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-11T05:09:45.949984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4415,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811616,"kb_used_data":1864,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675744,"statfs":{"total":386547056640,"available":383667961856,"internally_reserved":0,"allocated":1908736,"data_stored":746234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001125"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12903,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238269+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:33.238269+0000","last_peered":"2024-09-11T05:09:33.238269+0000","last_clean":"2024-09-11T05:09:33.238269+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:33.238269+0000","last_undegraded":"2024-09-11T05:09:33.238269+0000","last_fullsized":"2024-09-11T05:09:33.238269+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13970,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237645+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:33.237645+0000","last_peered":"2024-09-11T05:09:33.237645+0000","last_clean":"2024-09-11T05:09:33.237645+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:33.237645+0000","last_undegraded":"2024-09-11T05:09:33.237645+0000","last_fullsized":"2024-09-11T05:09:33.237645+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.236494+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:33.236494+0000","last_peered":"2024-09-11T05:09:33.236494+0000","last_clean":"2024-09-11T05:09:33.236494+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:33.236494+0000","last_undegraded":"2024-09-11T05:09:33.236494+0000","last_fullsized":"2024-09-11T05:09:33.236494+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17723,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237530+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:33.237530+0000","last_peered":"2024-09-11T05:09:33.237530+0000","last_clean":"2024-09-11T05:09:33.237530+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:33.237530+0000","last_undegraded":"2024-09-11T05:09:33.237530+0000","last_fullsized":"2024-09-11T05:09:33.237530+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":13814,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:45.403386+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:45.403386+0000","last_peered":"2024-09-11T05:09:45.403386+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:45.403386+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17027,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238337+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:33.238337+0000","last_peered":"2024-09-11T05:09:33.238337+0000","last_clean":"2024-09-11T05:09:33.238337+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:33.238337+0000","last_undegraded":"2024-09-11T05:09:33.238337+0000","last_fullsized":"2024-09-11T05:09:33.238337+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17541,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237785+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:33.237785+0000","last_peered":"2024-09-11T05:09:33.237785+0000","last_clean":"2024-09-11T05:09:33.237785+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:33.237785+0000","last_undegraded":"2024-09-11T05:09:33.237785+0000","last_fullsized":"2024-09-11T05:09:33.237785+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12737,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237828+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:33.237828+0000","last_peered":"2024-09-11T05:09:33.237828+0000","last_clean":"2024-09-11T05:09:33.237828+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:33.237828+0000","last_undegraded":"2024-09-11T05:09:33.237828+0000","last_fullsized":"2024-09-11T05:09:33.237828+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":15703,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:40.075790+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:40.075790+0000","last_peered":"2024-09-11T05:09:40.075790+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:40.075790+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-736}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12931,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237643+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:33.237643+0000","last_peered":"2024-09-11T05:09:33.237643+0000","last_clean":"2024-09-11T05:09:33.237643+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:33.237643+0000","last_undegraded":"2024-09-11T05:09:33.237643+0000","last_fullsized":"2024-09-11T05:09:33.237643+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4415,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623532,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748308,"statfs":{"total":96636764160,"available":95998267392,"internally_reserved":0,"allocated":487424,"data_stored":201822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:46.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:46.601 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:46.602 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:46.614 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-11T05:09:45.949984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4415,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811616,"kb_used_data":1864,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675744,"statfs":{"total":386547056640,"available":383667961856,"internally_reserved":0,"allocated":1908736,"data_stored":746234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001125"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12903,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238269+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:33.238269+0000","last_peered":"2024-09-11T05:09:33.238269+0000","last_clean":"2024-09-11T05:09:33.238269+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:33.238269+0000","last_undegraded":"2024-09-11T05:09:33.238269+0000","last_fullsized":"2024-09-11T05:09:33.238269+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13970,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237645+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:33.237645+0000","last_peered":"2024-09-11T05:09:33.237645+0000","last_clean":"2024-09-11T05:09:33.237645+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:33.237645+0000","last_undegraded":"2024-09-11T05:09:33.237645+0000","last_fullsized":"2024-09-11T05:09:33.237645+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.236494+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:33.236494+0000","last_peered":"2024-09-11T05:09:33.236494+0000","last_clean":"2024-09-11T05:09:33.236494+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:33.236494+0000","last_undegraded":"2024-09-11T05:09:33.236494+0000","last_fullsized":"2024-09-11T05:09:33.236494+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17723,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237530+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:33.237530+0000","last_peered":"2024-09-11T05:09:33.237530+0000","last_clean":"2024-09-11T05:09:33.237530+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:33.237530+0000","last_undegraded":"2024-09-11T05:09:33.237530+0000","last_fullsized":"2024-09-11T05:09:33.237530+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":13814,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:45.403386+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:45.403386+0000","last_peered":"2024-09-11T05:09:45.403386+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:45.403386+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17027,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238337+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:33.238337+0000","last_peered":"2024-09-11T05:09:33.238337+0000","last_clean":"2024-09-11T05:09:33.238337+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:33.238337+0000","last_undegraded":"2024-09-11T05:09:33.238337+0000","last_fullsized":"2024-09-11T05:09:33.238337+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17541,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237785+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:33.237785+0000","last_peered":"2024-09-11T05:09:33.237785+0000","last_clean":"2024-09-11T05:09:33.237785+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:33.237785+0000","last_undegraded":"2024-09-11T05:09:33.237785+0000","last_fullsized":"2024-09-11T05:09:33.237785+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12737,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237828+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:33.237828+0000","last_peered":"2024-09-11T05:09:33.237828+0000","last_clean":"2024-09-11T05:09:33.237828+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:33.237828+0000","last_undegraded":"2024-09-11T05:09:33.237828+0000","last_fullsized":"2024-09-11T05:09:33.237828+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":15703,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:40.075790+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:40.075790+0000","last_peered":"2024-09-11T05:09:40.075790+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:40.075790+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-736}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12931,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237643+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:33.237643+0000","last_peered":"2024-09-11T05:09:33.237643+0000","last_clean":"2024-09-11T05:09:33.237643+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:33.237643+0000","last_undegraded":"2024-09-11T05:09:33.237643+0000","last_fullsized":"2024-09-11T05:09:33.237643+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4415,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623532,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748308,"statfs":{"total":96636764160,"available":95998267392,"internally_reserved":0,"allocated":487424,"data_stored":201822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:46.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:46.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12655, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:44.269034+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:44.269034+0000', 'last_peered': '2024-09-11T05:09:44.269034+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:44.269034+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:46.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:46.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 13814, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:45.403386+0000', 'last_change': '2024-09-11T05:09:43.134714+0000', 'last_active': '2024-09-11T05:09:45.403386+0000', 'last_peered': '2024-09-11T05:09:45.403386+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:45.403386+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 461, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:46.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:46.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12813, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:44.269054+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:44.269054+0000', 'last_peered': '2024-09-11T05:09:44.269054+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:44.269054+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:46.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-11T05:09:46.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12407", 'reported_seq': 15703, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:40.075790+0000', 'last_change': '2024-09-11T05:09:31.310757+0000', 'last_active': '2024-09-11T05:09:40.075790+0000', 'last_peered': '2024-09-11T05:09:40.075790+0000', 'last_clean': '2024-09-11T04:55:08.668171+0000', 'last_became_active': '2024-09-11T04:56:20.645029+0000', 'last_became_peered': '2024-09-11T04:56:20.645029+0000', 'last_unstale': '2024-09-11T05:09:40.075790+0000', 'last_undegraded': '2024-09-11T04:56:20.250339+0000', 'last_fullsized': '2024-09-11T04:56:20.225433+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13472, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 738, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12405, 'num_write_kb': 49629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1584, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -736}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-11T05:09:46.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:47.056 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:47.056 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7269/36 objects degraded (20191.667%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":914,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":9,"data_bytes":60624,"bytes_used":2879094784,"bytes_avail":383667961856,"bytes_total":386547056640,"degraded_objects":7269,"degraded_total":36,"degraded_ratio":201.91666666666663,"recovering_objects_per_sec":168,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:47.071 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:09:47.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:47.414 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-11T05:09:45.949984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4415,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811616,"kb_used_data":1864,"kb_used_omap":14,"kb_used_meta":2809649,"kb_avail":374675744,"statfs":{"total":386547056640,"available":383667961856,"internally_reserved":0,"allocated":1908736,"data_stored":746234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2877081142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001125"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12903,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238269+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:33.238269+0000","last_peered":"2024-09-11T05:09:33.238269+0000","last_clean":"2024-09-11T05:09:33.238269+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:33.238269+0000","last_undegraded":"2024-09-11T05:09:33.238269+0000","last_fullsized":"2024-09-11T05:09:33.238269+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13970,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237645+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:33.237645+0000","last_peered":"2024-09-11T05:09:33.237645+0000","last_clean":"2024-09-11T05:09:33.237645+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:33.237645+0000","last_undegraded":"2024-09-11T05:09:33.237645+0000","last_fullsized":"2024-09-11T05:09:33.237645+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.236494+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:33.236494+0000","last_peered":"2024-09-11T05:09:33.236494+0000","last_clean":"2024-09-11T05:09:33.236494+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:33.236494+0000","last_undegraded":"2024-09-11T05:09:33.236494+0000","last_fullsized":"2024-09-11T05:09:33.236494+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17723,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237530+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:33.237530+0000","last_peered":"2024-09-11T05:09:33.237530+0000","last_clean":"2024-09-11T05:09:33.237530+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:33.237530+0000","last_undegraded":"2024-09-11T05:09:33.237530+0000","last_fullsized":"2024-09-11T05:09:33.237530+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":13814,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:45.403386+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:45.403386+0000","last_peered":"2024-09-11T05:09:45.403386+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:45.403386+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17027,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.238337+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:33.238337+0000","last_peered":"2024-09-11T05:09:33.238337+0000","last_clean":"2024-09-11T05:09:33.238337+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:33.238337+0000","last_undegraded":"2024-09-11T05:09:33.238337+0000","last_fullsized":"2024-09-11T05:09:33.238337+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17541,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237785+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:33.237785+0000","last_peered":"2024-09-11T05:09:33.237785+0000","last_clean":"2024-09-11T05:09:33.237785+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:33.237785+0000","last_undegraded":"2024-09-11T05:09:33.237785+0000","last_fullsized":"2024-09-11T05:09:33.237785+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12737,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237828+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:33.237828+0000","last_peered":"2024-09-11T05:09:33.237828+0000","last_clean":"2024-09-11T05:09:33.237828+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:33.237828+0000","last_undegraded":"2024-09-11T05:09:33.237828+0000","last_fullsized":"2024-09-11T05:09:33.237828+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":15703,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:40.075790+0000","last_change":"2024-09-11T05:09:31.310757+0000","last_active":"2024-09-11T05:09:40.075790+0000","last_peered":"2024-09-11T05:09:40.075790+0000","last_clean":"2024-09-11T04:55:08.668171+0000","last_became_active":"2024-09-11T04:56:20.645029+0000","last_became_peered":"2024-09-11T04:56:20.645029+0000","last_unstale":"2024-09-11T05:09:40.075790+0000","last_undegraded":"2024-09-11T04:56:20.250339+0000","last_fullsized":"2024-09-11T04:56:20.225433+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13472,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12405,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-736}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12931,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-11T05:09:33.237643+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:33.237643+0000","last_peered":"2024-09-11T05:09:33.237643+0000","last_clean":"2024-09-11T05:09:33.237643+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:33.237643+0000","last_undegraded":"2024-09-11T05:09:33.237643+0000","last_fullsized":"2024-09-11T05:09:33.237643+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":60624,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73583,"num_write_kb":252002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4415,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623532,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93748308,"statfs":{"total":96636764160,"available":95998267392,"internally_reserved":0,"allocated":487424,"data_stored":201822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":637988562},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:50.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:50.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:50.743 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:50.757 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-11T05:09:49.950706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819952,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667408,"statfs":{"total":386547056640,"available":383659425792,"internally_reserved":0,"allocated":1929216,"data_stored":746620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12405,"num_write_kb":-49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001276"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12907,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266738+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:44.266738+0000","last_peered":"2024-09-11T05:09:44.266738+0000","last_clean":"2024-09-11T05:09:44.266738+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:44.266738+0000","last_undegraded":"2024-09-11T05:09:44.266738+0000","last_fullsized":"2024-09-11T05:09:44.266738+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13974,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264902+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:44.264902+0000","last_peered":"2024-09-11T05:09:44.264902+0000","last_clean":"2024-09-11T05:09:44.264902+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:44.264902+0000","last_undegraded":"2024-09-11T05:09:44.264902+0000","last_fullsized":"2024-09-11T05:09:44.264902+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268849+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:44.268849+0000","last_peered":"2024-09-11T05:09:44.268849+0000","last_clean":"2024-09-11T05:09:44.268849+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:44.268849+0000","last_undegraded":"2024-09-11T05:09:44.268849+0000","last_fullsized":"2024-09-11T05:09:44.268849+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17727,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265143+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:44.265143+0000","last_peered":"2024-09-11T05:09:44.265143+0000","last_clean":"2024-09-11T05:09:44.265143+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:44.265143+0000","last_undegraded":"2024-09-11T05:09:44.265143+0000","last_fullsized":"2024-09-11T05:09:44.265143+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":13814,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:45.403386+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:45.403386+0000","last_peered":"2024-09-11T05:09:45.403386+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:45.403386+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17031,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:44.266724+0000","last_peered":"2024-09-11T05:09:44.266724+0000","last_clean":"2024-09-11T05:09:44.266724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:44.266724+0000","last_undegraded":"2024-09-11T05:09:44.266724+0000","last_fullsized":"2024-09-11T05:09:44.266724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17545,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265008+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:44.265008+0000","last_peered":"2024-09-11T05:09:44.265008+0000","last_clean":"2024-09-11T05:09:44.265008+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:44.265008+0000","last_undegraded":"2024-09-11T05:09:44.265008+0000","last_fullsized":"2024-09-11T05:09:44.265008+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12741,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265067+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:44.265067+0000","last_peered":"2024-09-11T05:09:44.265067+0000","last_clean":"2024-09-11T05:09:44.265067+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:44.265067+0000","last_undegraded":"2024-09-11T05:09:44.265067+0000","last_fullsized":"2024-09-11T05:09:44.265067+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12554,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.679590+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:44.679590+0000","last_peered":"2024-09-11T05:09:44.679590+0000","last_clean":"2024-09-11T05:09:44.679590+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:44.679590+0000","last_undegraded":"2024-09-11T05:09:44.679590+0000","last_fullsized":"2024-09-11T05:09:44.679590+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12935,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265248+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:44.265248+0000","last_peered":"2024-09-11T05:09:44.265248+0000","last_clean":"2024-09-11T05:09:44.265248+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:44.265248+0000","last_undegraded":"2024-09-11T05:09:44.265248+0000","last_fullsized":"2024-09-11T05:09:44.265248+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631856,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739984,"statfs":{"total":96636764160,"available":95989743616,"internally_reserved":0,"allocated":491520,"data_stored":195871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722088,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649752,"statfs":{"total":96636764160,"available":95897346048,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:50.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:51.070 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:51.071 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:51.083 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-11T05:09:49.950706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819952,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667408,"statfs":{"total":386547056640,"available":383659425792,"internally_reserved":0,"allocated":1929216,"data_stored":746620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12405,"num_write_kb":-49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001276"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12907,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266738+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:44.266738+0000","last_peered":"2024-09-11T05:09:44.266738+0000","last_clean":"2024-09-11T05:09:44.266738+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:44.266738+0000","last_undegraded":"2024-09-11T05:09:44.266738+0000","last_fullsized":"2024-09-11T05:09:44.266738+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13974,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264902+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:44.264902+0000","last_peered":"2024-09-11T05:09:44.264902+0000","last_clean":"2024-09-11T05:09:44.264902+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:44.264902+0000","last_undegraded":"2024-09-11T05:09:44.264902+0000","last_fullsized":"2024-09-11T05:09:44.264902+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268849+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:44.268849+0000","last_peered":"2024-09-11T05:09:44.268849+0000","last_clean":"2024-09-11T05:09:44.268849+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:44.268849+0000","last_undegraded":"2024-09-11T05:09:44.268849+0000","last_fullsized":"2024-09-11T05:09:44.268849+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17727,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265143+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:44.265143+0000","last_peered":"2024-09-11T05:09:44.265143+0000","last_clean":"2024-09-11T05:09:44.265143+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:44.265143+0000","last_undegraded":"2024-09-11T05:09:44.265143+0000","last_fullsized":"2024-09-11T05:09:44.265143+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":13814,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:45.403386+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:45.403386+0000","last_peered":"2024-09-11T05:09:45.403386+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:45.403386+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17031,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:44.266724+0000","last_peered":"2024-09-11T05:09:44.266724+0000","last_clean":"2024-09-11T05:09:44.266724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:44.266724+0000","last_undegraded":"2024-09-11T05:09:44.266724+0000","last_fullsized":"2024-09-11T05:09:44.266724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17545,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265008+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:44.265008+0000","last_peered":"2024-09-11T05:09:44.265008+0000","last_clean":"2024-09-11T05:09:44.265008+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:44.265008+0000","last_undegraded":"2024-09-11T05:09:44.265008+0000","last_fullsized":"2024-09-11T05:09:44.265008+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12741,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265067+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:44.265067+0000","last_peered":"2024-09-11T05:09:44.265067+0000","last_clean":"2024-09-11T05:09:44.265067+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:44.265067+0000","last_undegraded":"2024-09-11T05:09:44.265067+0000","last_fullsized":"2024-09-11T05:09:44.265067+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12554,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.679590+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:44.679590+0000","last_peered":"2024-09-11T05:09:44.679590+0000","last_clean":"2024-09-11T05:09:44.679590+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:44.679590+0000","last_undegraded":"2024-09-11T05:09:44.679590+0000","last_fullsized":"2024-09-11T05:09:44.679590+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12935,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265248+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:44.265248+0000","last_peered":"2024-09-11T05:09:44.265248+0000","last_clean":"2024-09-11T05:09:44.265248+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:44.265248+0000","last_undegraded":"2024-09-11T05:09:44.265248+0000","last_fullsized":"2024-09-11T05:09:44.265248+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631856,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739984,"statfs":{"total":96636764160,"available":95989743616,"internally_reserved":0,"allocated":491520,"data_stored":195871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722088,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649752,"statfs":{"total":96636764160,"available":95897346048,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:51.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:51.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12655, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:44.269034+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:44.269034+0000', 'last_peered': '2024-09-11T05:09:44.269034+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:44.269034+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:51.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:51.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 13814, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:45.403386+0000', 'last_change': '2024-09-11T05:09:43.134714+0000', 'last_active': '2024-09-11T05:09:45.403386+0000', 'last_peered': '2024-09-11T05:09:45.403386+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:45.403386+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 461, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:51.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:51.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12813, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:44.269054+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:44.269054+0000', 'last_peered': '2024-09-11T05:09:44.269054+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:44.269054+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:51.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:51.495 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6531/28 objects degraded (23325.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":918,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7,"data_bytes":47152,"bytes_used":2887630848,"bytes_avail":383659425792,"bytes_total":386547056640,"degraded_objects":6531,"degraded_total":28,"degraded_ratio":233.25},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:51.509 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:09:51.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:51.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:51.827 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:51.842 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-11T05:09:49.950706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819952,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2817969,"kb_avail":374667408,"statfs":{"total":386547056640,"available":383659425792,"internally_reserved":0,"allocated":1929216,"data_stored":746620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2885600822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12405,"num_write_kb":-49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001276"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12907,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266738+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:44.266738+0000","last_peered":"2024-09-11T05:09:44.266738+0000","last_clean":"2024-09-11T05:09:44.266738+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:44.266738+0000","last_undegraded":"2024-09-11T05:09:44.266738+0000","last_fullsized":"2024-09-11T05:09:44.266738+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13974,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264902+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:44.264902+0000","last_peered":"2024-09-11T05:09:44.264902+0000","last_clean":"2024-09-11T05:09:44.264902+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:44.264902+0000","last_undegraded":"2024-09-11T05:09:44.264902+0000","last_fullsized":"2024-09-11T05:09:44.264902+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268849+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:44.268849+0000","last_peered":"2024-09-11T05:09:44.268849+0000","last_clean":"2024-09-11T05:09:44.268849+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:44.268849+0000","last_undegraded":"2024-09-11T05:09:44.268849+0000","last_fullsized":"2024-09-11T05:09:44.268849+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17727,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265143+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:44.265143+0000","last_peered":"2024-09-11T05:09:44.265143+0000","last_clean":"2024-09-11T05:09:44.265143+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:44.265143+0000","last_undegraded":"2024-09-11T05:09:44.265143+0000","last_fullsized":"2024-09-11T05:09:44.265143+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":13814,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:45.403386+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:45.403386+0000","last_peered":"2024-09-11T05:09:45.403386+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:45.403386+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17031,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:44.266724+0000","last_peered":"2024-09-11T05:09:44.266724+0000","last_clean":"2024-09-11T05:09:44.266724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:44.266724+0000","last_undegraded":"2024-09-11T05:09:44.266724+0000","last_fullsized":"2024-09-11T05:09:44.266724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17545,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265008+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:44.265008+0000","last_peered":"2024-09-11T05:09:44.265008+0000","last_clean":"2024-09-11T05:09:44.265008+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:44.265008+0000","last_undegraded":"2024-09-11T05:09:44.265008+0000","last_fullsized":"2024-09-11T05:09:44.265008+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12741,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265067+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:44.265067+0000","last_peered":"2024-09-11T05:09:44.265067+0000","last_clean":"2024-09-11T05:09:44.265067+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:44.265067+0000","last_undegraded":"2024-09-11T05:09:44.265067+0000","last_fullsized":"2024-09-11T05:09:44.265067+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12554,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.679590+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:44.679590+0000","last_peered":"2024-09-11T05:09:44.679590+0000","last_clean":"2024-09-11T05:09:44.679590+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:44.679590+0000","last_undegraded":"2024-09-11T05:09:44.679590+0000","last_fullsized":"2024-09-11T05:09:44.679590+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12935,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265248+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:44.265248+0000","last_peered":"2024-09-11T05:09:44.265248+0000","last_clean":"2024-09-11T05:09:44.265248+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:44.265248+0000","last_undegraded":"2024-09-11T05:09:44.265248+0000","last_fullsized":"2024-09-11T05:09:44.265248+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631856,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93739984,"statfs":{"total":96636764160,"available":95989743616,"internally_reserved":0,"allocated":491520,"data_stored":195871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":646508242},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722088,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649752,"statfs":{"total":96636764160,"available":95897346048,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:54.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:55.200 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-11T05:09:53.951326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3640,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824112,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663248,"statfs":{"total":386547056640,"available":383655165952,"internally_reserved":0,"allocated":1929216,"data_stored":746620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12405,"num_write_kb":-49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-345,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001630"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12907,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266738+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:44.266738+0000","last_peered":"2024-09-11T05:09:44.266738+0000","last_clean":"2024-09-11T05:09:44.266738+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:44.266738+0000","last_undegraded":"2024-09-11T05:09:44.266738+0000","last_fullsized":"2024-09-11T05:09:44.266738+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13974,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264902+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:44.264902+0000","last_peered":"2024-09-11T05:09:44.264902+0000","last_clean":"2024-09-11T05:09:44.264902+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:44.264902+0000","last_undegraded":"2024-09-11T05:09:44.264902+0000","last_fullsized":"2024-09-11T05:09:44.264902+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268849+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:44.268849+0000","last_peered":"2024-09-11T05:09:44.268849+0000","last_clean":"2024-09-11T05:09:44.268849+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:44.268849+0000","last_undegraded":"2024-09-11T05:09:44.268849+0000","last_fullsized":"2024-09-11T05:09:44.268849+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17727,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265143+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:44.265143+0000","last_peered":"2024-09-11T05:09:44.265143+0000","last_clean":"2024-09-11T05:09:44.265143+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:44.265143+0000","last_undegraded":"2024-09-11T05:09:44.265143+0000","last_fullsized":"2024-09-11T05:09:44.265143+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":15432,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:50.406839+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:50.406839+0000","last_peered":"2024-09-11T05:09:50.406839+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:50.406839+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17031,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:44.266724+0000","last_peered":"2024-09-11T05:09:44.266724+0000","last_clean":"2024-09-11T05:09:44.266724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:44.266724+0000","last_undegraded":"2024-09-11T05:09:44.266724+0000","last_fullsized":"2024-09-11T05:09:44.266724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17545,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265008+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:44.265008+0000","last_peered":"2024-09-11T05:09:44.265008+0000","last_clean":"2024-09-11T05:09:44.265008+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:44.265008+0000","last_undegraded":"2024-09-11T05:09:44.265008+0000","last_fullsized":"2024-09-11T05:09:44.265008+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12741,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265067+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:44.265067+0000","last_peered":"2024-09-11T05:09:44.265067+0000","last_clean":"2024-09-11T05:09:44.265067+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:44.265067+0000","last_undegraded":"2024-09-11T05:09:44.265067+0000","last_fullsized":"2024-09-11T05:09:44.265067+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12554,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.679590+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:44.679590+0000","last_peered":"2024-09-11T05:09:44.679590+0000","last_clean":"2024-09-11T05:09:44.679590+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:44.679590+0000","last_undegraded":"2024-09-11T05:09:44.679590+0000","last_fullsized":"2024-09-11T05:09:44.679590+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12935,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265248+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:44.265248+0000","last_peered":"2024-09-11T05:09:44.265248+0000","last_clean":"2024-09-11T05:09:44.265248+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:44.265248+0000","last_undegraded":"2024-09-11T05:09:44.265248+0000","last_fullsized":"2024-09-11T05:09:44.265248+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3640,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636016,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735824,"statfs":{"total":96636764160,"available":95985483776,"internally_reserved":0,"allocated":491520,"data_stored":195871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722088,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649752,"statfs":{"total":96636764160,"available":95897346048,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:55.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:55.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:55.517 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:55.529 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-11T05:09:53.951326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3640,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824112,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663248,"statfs":{"total":386547056640,"available":383655165952,"internally_reserved":0,"allocated":1929216,"data_stored":746620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12405,"num_write_kb":-49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-345,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001630"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12907,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266738+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:44.266738+0000","last_peered":"2024-09-11T05:09:44.266738+0000","last_clean":"2024-09-11T05:09:44.266738+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:44.266738+0000","last_undegraded":"2024-09-11T05:09:44.266738+0000","last_fullsized":"2024-09-11T05:09:44.266738+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13974,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264902+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:44.264902+0000","last_peered":"2024-09-11T05:09:44.264902+0000","last_clean":"2024-09-11T05:09:44.264902+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:44.264902+0000","last_undegraded":"2024-09-11T05:09:44.264902+0000","last_fullsized":"2024-09-11T05:09:44.264902+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268849+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:44.268849+0000","last_peered":"2024-09-11T05:09:44.268849+0000","last_clean":"2024-09-11T05:09:44.268849+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:44.268849+0000","last_undegraded":"2024-09-11T05:09:44.268849+0000","last_fullsized":"2024-09-11T05:09:44.268849+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17727,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265143+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:44.265143+0000","last_peered":"2024-09-11T05:09:44.265143+0000","last_clean":"2024-09-11T05:09:44.265143+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:44.265143+0000","last_undegraded":"2024-09-11T05:09:44.265143+0000","last_fullsized":"2024-09-11T05:09:44.265143+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":15432,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:50.406839+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:50.406839+0000","last_peered":"2024-09-11T05:09:50.406839+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:50.406839+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17031,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:44.266724+0000","last_peered":"2024-09-11T05:09:44.266724+0000","last_clean":"2024-09-11T05:09:44.266724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:44.266724+0000","last_undegraded":"2024-09-11T05:09:44.266724+0000","last_fullsized":"2024-09-11T05:09:44.266724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17545,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265008+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:44.265008+0000","last_peered":"2024-09-11T05:09:44.265008+0000","last_clean":"2024-09-11T05:09:44.265008+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:44.265008+0000","last_undegraded":"2024-09-11T05:09:44.265008+0000","last_fullsized":"2024-09-11T05:09:44.265008+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12741,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265067+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:44.265067+0000","last_peered":"2024-09-11T05:09:44.265067+0000","last_clean":"2024-09-11T05:09:44.265067+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:44.265067+0000","last_undegraded":"2024-09-11T05:09:44.265067+0000","last_fullsized":"2024-09-11T05:09:44.265067+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12554,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.679590+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:44.679590+0000","last_peered":"2024-09-11T05:09:44.679590+0000","last_clean":"2024-09-11T05:09:44.679590+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:44.679590+0000","last_undegraded":"2024-09-11T05:09:44.679590+0000","last_fullsized":"2024-09-11T05:09:44.679590+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12935,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265248+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:44.265248+0000","last_peered":"2024-09-11T05:09:44.265248+0000","last_clean":"2024-09-11T05:09:44.265248+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:44.265248+0000","last_undegraded":"2024-09-11T05:09:44.265248+0000","last_fullsized":"2024-09-11T05:09:44.265248+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3640,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636016,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735824,"statfs":{"total":96636764160,"available":95985483776,"internally_reserved":0,"allocated":491520,"data_stored":195871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722088,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649752,"statfs":{"total":96636764160,"available":95897346048,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:55.530 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:55.530 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12655, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:44.269034+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:44.269034+0000', 'last_peered': '2024-09-11T05:09:44.269034+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:44.269034+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:55.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:55.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 15432, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:50.406839+0000', 'last_change': '2024-09-11T05:09:43.134714+0000', 'last_active': '2024-09-11T05:09:50.406839+0000', 'last_peered': '2024-09-11T05:09:50.406839+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:50.406839+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1270, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1126}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:55.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:55.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12813, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:44.269054+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:44.269054+0000', 'last_peered': '2024-09-11T05:09:44.269054+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:44.269054+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:55.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:09:55.910 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:55.910 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5722/28 objects degraded (20435.714%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":922,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7,"data_bytes":47152,"bytes_used":2891890688,"bytes_avail":383655165952,"bytes_total":386547056640,"degraded_objects":5722,"degraded_total":28,"degraded_ratio":204.35714285714286},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:09:55.923 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:09:55.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:56.242 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:56.255 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-11T05:09:55.951637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824112,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663248,"statfs":{"total":386547056640,"available":383655165952,"internally_reserved":0,"allocated":1929216,"data_stored":746620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13472,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12405,"num_write_kb":-49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.688351"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12907,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266738+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:44.266738+0000","last_peered":"2024-09-11T05:09:44.266738+0000","last_clean":"2024-09-11T05:09:44.266738+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:44.266738+0000","last_undegraded":"2024-09-11T05:09:44.266738+0000","last_fullsized":"2024-09-11T05:09:44.266738+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13974,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264902+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:44.264902+0000","last_peered":"2024-09-11T05:09:44.264902+0000","last_clean":"2024-09-11T05:09:44.264902+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:44.264902+0000","last_undegraded":"2024-09-11T05:09:44.264902+0000","last_fullsized":"2024-09-11T05:09:44.264902+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268849+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:44.268849+0000","last_peered":"2024-09-11T05:09:44.268849+0000","last_clean":"2024-09-11T05:09:44.268849+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:44.268849+0000","last_undegraded":"2024-09-11T05:09:44.268849+0000","last_fullsized":"2024-09-11T05:09:44.268849+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17727,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265143+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:44.265143+0000","last_peered":"2024-09-11T05:09:44.265143+0000","last_clean":"2024-09-11T05:09:44.265143+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:44.265143+0000","last_undegraded":"2024-09-11T05:09:44.265143+0000","last_fullsized":"2024-09-11T05:09:44.265143+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17580,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:55.409203+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:55.409203+0000","last_peered":"2024-09-11T05:09:55.409203+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:55.409203+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17031,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.266724+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:44.266724+0000","last_peered":"2024-09-11T05:09:44.266724+0000","last_clean":"2024-09-11T05:09:44.266724+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:44.266724+0000","last_undegraded":"2024-09-11T05:09:44.266724+0000","last_fullsized":"2024-09-11T05:09:44.266724+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17545,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265008+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:44.265008+0000","last_peered":"2024-09-11T05:09:44.265008+0000","last_clean":"2024-09-11T05:09:44.265008+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:44.265008+0000","last_undegraded":"2024-09-11T05:09:44.265008+0000","last_fullsized":"2024-09-11T05:09:44.265008+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12741,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265067+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:44.265067+0000","last_peered":"2024-09-11T05:09:44.265067+0000","last_clean":"2024-09-11T05:09:44.265067+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:44.265067+0000","last_undegraded":"2024-09-11T05:09:44.265067+0000","last_fullsized":"2024-09-11T05:09:44.265067+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12554,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.679590+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:44.679590+0000","last_peered":"2024-09-11T05:09:44.679590+0000","last_clean":"2024-09-11T05:09:44.679590+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:44.679590+0000","last_undegraded":"2024-09-11T05:09:44.679590+0000","last_fullsized":"2024-09-11T05:09:44.679590+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12935,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.265248+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:44.265248+0000","last_peered":"2024-09-11T05:09:44.265248+0000","last_clean":"2024-09-11T05:09:44.265248+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:44.265248+0000","last_undegraded":"2024-09-11T05:09:44.265248+0000","last_fullsized":"2024-09-11T05:09:44.265248+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636016,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735824,"statfs":{"total":96636764160,"available":95985483776,"internally_reserved":0,"allocated":491520,"data_stored":195871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722088,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649752,"statfs":{"total":96636764160,"available":95897346048,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:59.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:59.594 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:59.609 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-11T05:09:57.952008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815812,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671548,"statfs":{"total":386547056640,"available":383663665152,"internally_reserved":0,"allocated":1945600,"data_stored":752904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1883,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001301"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12909,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311383+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:56.311383+0000","last_peered":"2024-09-11T05:09:56.311383+0000","last_clean":"2024-09-11T05:09:56.311383+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:56.311383+0000","last_undegraded":"2024-09-11T05:09:56.311383+0000","last_fullsized":"2024-09-11T05:09:56.311383+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13976,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310014+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:56.310014+0000","last_peered":"2024-09-11T05:09:56.310014+0000","last_clean":"2024-09-11T05:09:56.310014+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:56.310014+0000","last_undegraded":"2024-09-11T05:09:56.310014+0000","last_fullsized":"2024-09-11T05:09:56.310014+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311484+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311484+0000","last_peered":"2024-09-11T05:09:57.311484+0000","last_clean":"2024-09-11T05:09:57.311484+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311484+0000","last_undegraded":"2024-09-11T05:09:57.311484+0000","last_fullsized":"2024-09-11T05:09:57.311484+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17729,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.309961+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:56.309961+0000","last_peered":"2024-09-11T05:09:56.309961+0000","last_clean":"2024-09-11T05:09:56.309961+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:56.309961+0000","last_undegraded":"2024-09-11T05:09:56.309961+0000","last_fullsized":"2024-09-11T05:09:56.309961+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17580,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:55.409203+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:55.409203+0000","last_peered":"2024-09-11T05:09:55.409203+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:55.409203+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17033,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311540+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:56.311540+0000","last_peered":"2024-09-11T05:09:56.311540+0000","last_clean":"2024-09-11T05:09:56.311540+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:56.311540+0000","last_undegraded":"2024-09-11T05:09:56.311540+0000","last_fullsized":"2024-09-11T05:09:56.311540+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17547,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.309956+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:56.309956+0000","last_peered":"2024-09-11T05:09:56.309956+0000","last_clean":"2024-09-11T05:09:56.309956+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:56.309956+0000","last_undegraded":"2024-09-11T05:09:56.309956+0000","last_fullsized":"2024-09-11T05:09:56.309956+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12743,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310527+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:56.310527+0000","last_peered":"2024-09-11T05:09:56.310527+0000","last_clean":"2024-09-11T05:09:56.310527+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:56.310527+0000","last_undegraded":"2024-09-11T05:09:56.310527+0000","last_fullsized":"2024-09-11T05:09:56.310527+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12556,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311337+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:56.311337+0000","last_peered":"2024-09-11T05:09:56.311337+0000","last_clean":"2024-09-11T05:09:56.311337+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:56.311337+0000","last_undegraded":"2024-09-11T05:09:56.311337+0000","last_fullsized":"2024-09-11T05:09:56.311337+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12937,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310204+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:56.310204+0000","last_peered":"2024-09-11T05:09:56.310204+0000","last_clean":"2024-09-11T05:09:56.310204+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:56.310204+0000","last_undegraded":"2024-09-11T05:09:56.310204+0000","last_fullsized":"2024-09-11T05:09:56.310204+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627708,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744132,"statfs":{"total":96636764160,"available":95993991168,"internally_reserved":0,"allocated":499712,"data_stored":199013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722096,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649744,"statfs":{"total":96636764160,"available":95897337856,"internally_reserved":0,"allocated":487424,"data_stored":186725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:59.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:09:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:09:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:09:59.939 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-11T05:09:57.952008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815812,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671548,"statfs":{"total":386547056640,"available":383663665152,"internally_reserved":0,"allocated":1945600,"data_stored":752904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1883,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001301"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12909,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311383+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:56.311383+0000","last_peered":"2024-09-11T05:09:56.311383+0000","last_clean":"2024-09-11T05:09:56.311383+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:56.311383+0000","last_undegraded":"2024-09-11T05:09:56.311383+0000","last_fullsized":"2024-09-11T05:09:56.311383+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13976,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310014+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:56.310014+0000","last_peered":"2024-09-11T05:09:56.310014+0000","last_clean":"2024-09-11T05:09:56.310014+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:56.310014+0000","last_undegraded":"2024-09-11T05:09:56.310014+0000","last_fullsized":"2024-09-11T05:09:56.310014+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311484+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311484+0000","last_peered":"2024-09-11T05:09:57.311484+0000","last_clean":"2024-09-11T05:09:57.311484+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311484+0000","last_undegraded":"2024-09-11T05:09:57.311484+0000","last_fullsized":"2024-09-11T05:09:57.311484+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17729,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.309961+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:56.309961+0000","last_peered":"2024-09-11T05:09:56.309961+0000","last_clean":"2024-09-11T05:09:56.309961+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:56.309961+0000","last_undegraded":"2024-09-11T05:09:56.309961+0000","last_fullsized":"2024-09-11T05:09:56.309961+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17580,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:55.409203+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:55.409203+0000","last_peered":"2024-09-11T05:09:55.409203+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:55.409203+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17033,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311540+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:56.311540+0000","last_peered":"2024-09-11T05:09:56.311540+0000","last_clean":"2024-09-11T05:09:56.311540+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:56.311540+0000","last_undegraded":"2024-09-11T05:09:56.311540+0000","last_fullsized":"2024-09-11T05:09:56.311540+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17547,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.309956+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:56.309956+0000","last_peered":"2024-09-11T05:09:56.309956+0000","last_clean":"2024-09-11T05:09:56.309956+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:56.309956+0000","last_undegraded":"2024-09-11T05:09:56.309956+0000","last_fullsized":"2024-09-11T05:09:56.309956+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12743,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310527+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:56.310527+0000","last_peered":"2024-09-11T05:09:56.310527+0000","last_clean":"2024-09-11T05:09:56.310527+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:56.310527+0000","last_undegraded":"2024-09-11T05:09:56.310527+0000","last_fullsized":"2024-09-11T05:09:56.310527+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12556,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311337+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:56.311337+0000","last_peered":"2024-09-11T05:09:56.311337+0000","last_clean":"2024-09-11T05:09:56.311337+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:56.311337+0000","last_undegraded":"2024-09-11T05:09:56.311337+0000","last_fullsized":"2024-09-11T05:09:56.311337+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12937,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310204+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:56.310204+0000","last_peered":"2024-09-11T05:09:56.310204+0000","last_clean":"2024-09-11T05:09:56.310204+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:56.310204+0000","last_undegraded":"2024-09-11T05:09:56.310204+0000","last_fullsized":"2024-09-11T05:09:56.310204+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627708,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744132,"statfs":{"total":96636764160,"available":95993991168,"internally_reserved":0,"allocated":499712,"data_stored":199013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722096,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649744,"statfs":{"total":96636764160,"available":95897337856,"internally_reserved":0,"allocated":487424,"data_stored":186725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:09:59.940 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:09:59.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 12655, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:44.269034+0000', 'last_change': '2024-09-11T04:56:20.629800+0000', 'last_active': '2024-09-11T05:09:44.269034+0000', 'last_peered': '2024-09-11T05:09:44.269034+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:09:44.269034+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:59.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-11T05:09:59.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12784", 'reported_seq': 17580, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:55.409203+0000', 'last_change': '2024-09-11T05:09:43.134714+0000', 'last_active': '2024-09-11T05:09:55.409203+0000', 'last_peered': '2024-09-11T05:09:55.409203+0000', 'last_clean': '2024-09-11T04:55:08.350284+0000', 'last_became_active': '2024-09-11T04:56:20.612938+0000', 'last_became_peered': '2024-09-11T04:56:20.612938+0000', 'last_unstale': '2024-09-11T05:09:55.409203+0000', 'last_undegraded': '2024-09-11T04:56:20.316958+0000', 'last_fullsized': '2024-09-11T04:56:20.277136+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26944, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 56, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12781, 'num_write_kb': 51137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2344, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -52}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:59.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:09:59.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12813, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:44.269054+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:44.269054+0000', 'last_peered': '2024-09-11T05:09:44.269054+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:44.269054+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:09:59.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:10:00.400 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:00.400 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4648/28 objects degraded (16600.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":927,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7,"data_bytes":47152,"bytes_used":2883391488,"bytes_avail":383663665152,"bytes_total":386547056640,"degraded_objects":4648,"degraded_total":28,"degraded_ratio":166,"recovering_objects_per_sec":134,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1074,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:10:00.414 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:10:00.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:00.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:00.744 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:00.756 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-09-11T05:09:59.952324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815812,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671548,"statfs":{"total":386547056640,"available":383663665152,"internally_reserved":0,"allocated":1945600,"data_stored":752904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1074,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001343"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":12655,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269034+0000","last_change":"2024-09-11T04:56:20.629800+0000","last_active":"2024-09-11T05:09:44.269034+0000","last_peered":"2024-09-11T05:09:44.269034+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:09:44.269034+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12909,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311383+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:56.311383+0000","last_peered":"2024-09-11T05:09:56.311383+0000","last_clean":"2024-09-11T05:09:56.311383+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:56.311383+0000","last_undegraded":"2024-09-11T05:09:56.311383+0000","last_fullsized":"2024-09-11T05:09:56.311383+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16753,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264647+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:44.264647+0000","last_peered":"2024-09-11T05:09:44.264647+0000","last_clean":"2024-09-11T05:09:44.264647+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:44.264647+0000","last_undegraded":"2024-09-11T05:09:44.264647+0000","last_fullsized":"2024-09-11T05:09:44.264647+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13976,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310014+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:56.310014+0000","last_peered":"2024-09-11T05:09:56.310014+0000","last_clean":"2024-09-11T05:09:56.310014+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:56.310014+0000","last_undegraded":"2024-09-11T05:09:56.310014+0000","last_fullsized":"2024-09-11T05:09:56.310014+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17163,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.268814+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:44.268814+0000","last_peered":"2024-09-11T05:09:44.268814+0000","last_clean":"2024-09-11T05:09:44.268814+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:44.268814+0000","last_undegraded":"2024-09-11T05:09:44.268814+0000","last_fullsized":"2024-09-11T05:09:44.268814+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17390,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.271998+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:44.271998+0000","last_peered":"2024-09-11T05:09:44.271998+0000","last_clean":"2024-09-11T05:09:44.271998+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:44.271998+0000","last_undegraded":"2024-09-11T05:09:44.271998+0000","last_fullsized":"2024-09-11T05:09:44.271998+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311484+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311484+0000","last_peered":"2024-09-11T05:09:57.311484+0000","last_clean":"2024-09-11T05:09:57.311484+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311484+0000","last_undegraded":"2024-09-11T05:09:57.311484+0000","last_fullsized":"2024-09-11T05:09:57.311484+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17729,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.309961+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:56.309961+0000","last_peered":"2024-09-11T05:09:56.309961+0000","last_clean":"2024-09-11T05:09:56.309961+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:56.309961+0000","last_undegraded":"2024-09-11T05:09:56.309961+0000","last_fullsized":"2024-09-11T05:09:56.309961+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17580,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:55.409203+0000","last_change":"2024-09-11T05:09:43.134714+0000","last_active":"2024-09-11T05:09:55.409203+0000","last_peered":"2024-09-11T05:09:55.409203+0000","last_clean":"2024-09-11T04:55:08.350284+0000","last_became_active":"2024-09-11T04:56:20.612938+0000","last_became_peered":"2024-09-11T04:56:20.612938+0000","last_unstale":"2024-09-11T05:09:55.409203+0000","last_undegraded":"2024-09-11T04:56:20.316958+0000","last_fullsized":"2024-09-11T04:56:20.277136+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-52}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17319,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T05:09:44.264773+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:44.264773+0000","last_peered":"2024-09-11T05:09:44.264773+0000","last_clean":"2024-09-11T05:09:44.264773+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:44.264773+0000","last_undegraded":"2024-09-11T05:09:44.264773+0000","last_fullsized":"2024-09-11T05:09:44.264773+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12813,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:44.269054+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:44.269054+0000","last_peered":"2024-09-11T05:09:44.269054+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:44.269054+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17033,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311540+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:56.311540+0000","last_peered":"2024-09-11T05:09:56.311540+0000","last_clean":"2024-09-11T05:09:56.311540+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:56.311540+0000","last_undegraded":"2024-09-11T05:09:56.311540+0000","last_fullsized":"2024-09-11T05:09:56.311540+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17547,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.309956+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:56.309956+0000","last_peered":"2024-09-11T05:09:56.309956+0000","last_clean":"2024-09-11T05:09:56.309956+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:56.309956+0000","last_undegraded":"2024-09-11T05:09:56.309956+0000","last_fullsized":"2024-09-11T05:09:56.309956+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12743,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310527+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:56.310527+0000","last_peered":"2024-09-11T05:09:56.310527+0000","last_clean":"2024-09-11T05:09:56.310527+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:56.310527+0000","last_undegraded":"2024-09-11T05:09:56.310527+0000","last_fullsized":"2024-09-11T05:09:56.310527+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12556,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.311337+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:56.311337+0000","last_peered":"2024-09-11T05:09:56.311337+0000","last_clean":"2024-09-11T05:09:56.311337+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:56.311337+0000","last_undegraded":"2024-09-11T05:09:56.311337+0000","last_fullsized":"2024-09-11T05:09:56.311337+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12937,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T05:09:56.310204+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:56.310204+0000","last_peered":"2024-09-11T05:09:56.310204+0000","last_clean":"2024-09-11T05:09:56.310204+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:56.310204+0000","last_undegraded":"2024-09-11T05:09:56.310204+0000","last_fullsized":"2024-09-11T05:09:56.310204+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47152,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61178,"num_write_kb":202373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744364,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627476,"statfs":{"total":96636764160,"available":95874535424,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721644,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650196,"statfs":{"total":96636764160,"available":95897800704,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627708,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744132,"statfs":{"total":96636764160,"available":95993991168,"internally_reserved":0,"allocated":499712,"data_stored":199013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722096,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649744,"statfs":{"total":96636764160,"available":95897337856,"internally_reserved":0,"allocated":487424,"data_stored":186725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:03.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:04.068 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:04.081 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-09-11T05:10:03.953030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824180,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663180,"statfs":{"total":386547056640,"available":383655096320,"internally_reserved":0,"allocated":1994752,"data_stored":771672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12781,"num_write_kb":-51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001393"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":14220,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:00.405620+0000","last_change":"2024-09-11T05:09:55.662190+0000","last_active":"2024-09-11T05:10:00.405620+0000","last_peered":"2024-09-11T05:10:00.405620+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:10:00.405620+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1511}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12911,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307526+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:57.307526+0000","last_peered":"2024-09-11T05:09:57.307526+0000","last_clean":"2024-09-11T05:09:57.307526+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:57.307526+0000","last_undegraded":"2024-09-11T05:09:57.307526+0000","last_fullsized":"2024-09-11T05:09:57.307526+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16757,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308709+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:57.308709+0000","last_peered":"2024-09-11T05:09:57.308709+0000","last_clean":"2024-09-11T05:09:57.308709+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:57.308709+0000","last_undegraded":"2024-09-11T05:09:57.308709+0000","last_fullsized":"2024-09-11T05:09:57.308709+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13979,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306716+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:57.306716+0000","last_peered":"2024-09-11T05:09:57.306716+0000","last_clean":"2024-09-11T05:09:57.306716+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:57.306716+0000","last_undegraded":"2024-09-11T05:09:57.306716+0000","last_fullsized":"2024-09-11T05:09:57.306716+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17167,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308998+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:57.308998+0000","last_peered":"2024-09-11T05:09:57.308998+0000","last_clean":"2024-09-11T05:09:57.308998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:57.308998+0000","last_undegraded":"2024-09-11T05:09:57.308998+0000","last_fullsized":"2024-09-11T05:09:57.308998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17394,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.309470+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:57.309470+0000","last_peered":"2024-09-11T05:09:57.309470+0000","last_clean":"2024-09-11T05:09:57.309470+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:57.309470+0000","last_undegraded":"2024-09-11T05:09:57.309470+0000","last_fullsized":"2024-09-11T05:09:57.309470+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17509,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311539+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311539+0000","last_peered":"2024-09-11T05:09:57.311539+0000","last_clean":"2024-09-11T05:09:57.311539+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311539+0000","last_undegraded":"2024-09-11T05:09:57.311539+0000","last_fullsized":"2024-09-11T05:09:57.311539+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17731,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307005+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:57.307005+0000","last_peered":"2024-09-11T05:09:57.307005+0000","last_clean":"2024-09-11T05:09:57.307005+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:57.307005+0000","last_undegraded":"2024-09-11T05:09:57.307005+0000","last_fullsized":"2024-09-11T05:09:57.307005+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17713,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.409213+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:09:57.409213+0000","last_peered":"2024-09-11T05:09:57.409213+0000","last_clean":"2024-09-11T05:09:57.409213+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:09:57.409213+0000","last_undegraded":"2024-09-11T05:09:57.409213+0000","last_fullsized":"2024-09-11T05:09:57.409213+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17323,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308493+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:57.308493+0000","last_peered":"2024-09-11T05:09:57.308493+0000","last_clean":"2024-09-11T05:09:57.308493+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:57.308493+0000","last_undegraded":"2024-09-11T05:09:57.308493+0000","last_fullsized":"2024-09-11T05:09:57.308493+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12817,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:57.309063+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:57.309063+0000","last_peered":"2024-09-11T05:09:57.309063+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:57.309063+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17035,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307639+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:57.307639+0000","last_peered":"2024-09-11T05:09:57.307639+0000","last_clean":"2024-09-11T05:09:57.307639+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:57.307639+0000","last_undegraded":"2024-09-11T05:09:57.307639+0000","last_fullsized":"2024-09-11T05:09:57.307639+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17549,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307146+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:57.307146+0000","last_peered":"2024-09-11T05:09:57.307146+0000","last_clean":"2024-09-11T05:09:57.307146+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:57.307146+0000","last_undegraded":"2024-09-11T05:09:57.307146+0000","last_fullsized":"2024-09-11T05:09:57.307146+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12745,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306929+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:57.306929+0000","last_peered":"2024-09-11T05:09:57.306929+0000","last_clean":"2024-09-11T05:09:57.306929+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:57.306929+0000","last_undegraded":"2024-09-11T05:09:57.306929+0000","last_fullsized":"2024-09-11T05:09:57.306929+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307165+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:57.307165+0000","last_peered":"2024-09-11T05:09:57.307165+0000","last_clean":"2024-09-11T05:09:57.307165+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:57.307165+0000","last_undegraded":"2024-09-11T05:09:57.307165+0000","last_fullsized":"2024-09-11T05:09:57.307165+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307037+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:57.307037+0000","last_peered":"2024-09-11T05:09:57.307037+0000","last_clean":"2024-09-11T05:09:57.307037+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:57.307037+0000","last_undegraded":"2024-09-11T05:09:57.307037+0000","last_fullsized":"2024-09-11T05:09:57.307037+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744380,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627460,"statfs":{"total":96636764160,"available":95874519040,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721660,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650180,"statfs":{"total":96636764160,"available":95897784320,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636032,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735808,"statfs":{"total":96636764160,"available":95985467392,"internally_reserved":0,"allocated":507904,"data_stored":202134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:04.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:04.394 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:04.394 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:04.406 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-09-11T05:10:03.953030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824180,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663180,"statfs":{"total":386547056640,"available":383655096320,"internally_reserved":0,"allocated":1994752,"data_stored":771672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12781,"num_write_kb":-51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001393"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":14220,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:00.405620+0000","last_change":"2024-09-11T05:09:55.662190+0000","last_active":"2024-09-11T05:10:00.405620+0000","last_peered":"2024-09-11T05:10:00.405620+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:10:00.405620+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1511}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12911,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307526+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:57.307526+0000","last_peered":"2024-09-11T05:09:57.307526+0000","last_clean":"2024-09-11T05:09:57.307526+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:57.307526+0000","last_undegraded":"2024-09-11T05:09:57.307526+0000","last_fullsized":"2024-09-11T05:09:57.307526+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16757,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308709+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:57.308709+0000","last_peered":"2024-09-11T05:09:57.308709+0000","last_clean":"2024-09-11T05:09:57.308709+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:57.308709+0000","last_undegraded":"2024-09-11T05:09:57.308709+0000","last_fullsized":"2024-09-11T05:09:57.308709+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13979,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306716+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:57.306716+0000","last_peered":"2024-09-11T05:09:57.306716+0000","last_clean":"2024-09-11T05:09:57.306716+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:57.306716+0000","last_undegraded":"2024-09-11T05:09:57.306716+0000","last_fullsized":"2024-09-11T05:09:57.306716+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17167,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308998+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:57.308998+0000","last_peered":"2024-09-11T05:09:57.308998+0000","last_clean":"2024-09-11T05:09:57.308998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:57.308998+0000","last_undegraded":"2024-09-11T05:09:57.308998+0000","last_fullsized":"2024-09-11T05:09:57.308998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17394,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.309470+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:57.309470+0000","last_peered":"2024-09-11T05:09:57.309470+0000","last_clean":"2024-09-11T05:09:57.309470+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:57.309470+0000","last_undegraded":"2024-09-11T05:09:57.309470+0000","last_fullsized":"2024-09-11T05:09:57.309470+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17509,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311539+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311539+0000","last_peered":"2024-09-11T05:09:57.311539+0000","last_clean":"2024-09-11T05:09:57.311539+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311539+0000","last_undegraded":"2024-09-11T05:09:57.311539+0000","last_fullsized":"2024-09-11T05:09:57.311539+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17731,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307005+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:57.307005+0000","last_peered":"2024-09-11T05:09:57.307005+0000","last_clean":"2024-09-11T05:09:57.307005+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:57.307005+0000","last_undegraded":"2024-09-11T05:09:57.307005+0000","last_fullsized":"2024-09-11T05:09:57.307005+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17713,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.409213+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:09:57.409213+0000","last_peered":"2024-09-11T05:09:57.409213+0000","last_clean":"2024-09-11T05:09:57.409213+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:09:57.409213+0000","last_undegraded":"2024-09-11T05:09:57.409213+0000","last_fullsized":"2024-09-11T05:09:57.409213+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17323,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308493+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:57.308493+0000","last_peered":"2024-09-11T05:09:57.308493+0000","last_clean":"2024-09-11T05:09:57.308493+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:57.308493+0000","last_undegraded":"2024-09-11T05:09:57.308493+0000","last_fullsized":"2024-09-11T05:09:57.308493+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12817,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:57.309063+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:57.309063+0000","last_peered":"2024-09-11T05:09:57.309063+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:57.309063+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17035,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307639+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:57.307639+0000","last_peered":"2024-09-11T05:09:57.307639+0000","last_clean":"2024-09-11T05:09:57.307639+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:57.307639+0000","last_undegraded":"2024-09-11T05:09:57.307639+0000","last_fullsized":"2024-09-11T05:09:57.307639+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17549,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307146+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:57.307146+0000","last_peered":"2024-09-11T05:09:57.307146+0000","last_clean":"2024-09-11T05:09:57.307146+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:57.307146+0000","last_undegraded":"2024-09-11T05:09:57.307146+0000","last_fullsized":"2024-09-11T05:09:57.307146+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12745,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306929+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:57.306929+0000","last_peered":"2024-09-11T05:09:57.306929+0000","last_clean":"2024-09-11T05:09:57.306929+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:57.306929+0000","last_undegraded":"2024-09-11T05:09:57.306929+0000","last_fullsized":"2024-09-11T05:09:57.306929+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307165+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:57.307165+0000","last_peered":"2024-09-11T05:09:57.307165+0000","last_clean":"2024-09-11T05:09:57.307165+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:57.307165+0000","last_undegraded":"2024-09-11T05:09:57.307165+0000","last_fullsized":"2024-09-11T05:09:57.307165+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307037+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:57.307037+0000","last_peered":"2024-09-11T05:09:57.307037+0000","last_clean":"2024-09-11T05:09:57.307037+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:57.307037+0000","last_undegraded":"2024-09-11T05:09:57.307037+0000","last_fullsized":"2024-09-11T05:09:57.307037+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744380,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627460,"statfs":{"total":96636764160,"available":95874519040,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721660,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650180,"statfs":{"total":96636764160,"available":95897784320,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636032,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735808,"statfs":{"total":96636764160,"available":95985467392,"internally_reserved":0,"allocated":507904,"data_stored":202134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:04.407 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:10:04.407 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 14220, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:10:00.405620+0000', 'last_change': '2024-09-11T05:09:55.662190+0000', 'last_active': '2024-09-11T05:10:00.405620+0000', 'last_peered': '2024-09-11T05:10:00.405620+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:10:00.405620+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1511, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 801, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1511}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:10:04.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:10:04.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12817, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:57.309063+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:57.309063+0000', 'last_peered': '2024-09-11T05:09:57.309063+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:57.309063+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:10:04.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:10:04.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:04.835 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3812/16 objects degraded (23825.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":931,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":26944,"bytes_used":2891960320,"bytes_avail":383655096320,"bytes_total":386547056640,"degraded_objects":3812,"degraded_total":16,"degraded_ratio":238.25},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:10:04.848 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:10:04.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:05.168 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:05.168 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:05.181 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-09-11T05:10:03.953030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824180,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663180,"statfs":{"total":386547056640,"available":383655096320,"internally_reserved":0,"allocated":1994752,"data_stored":771672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12781,"num_write_kb":-51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001393"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":14220,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:00.405620+0000","last_change":"2024-09-11T05:09:55.662190+0000","last_active":"2024-09-11T05:10:00.405620+0000","last_peered":"2024-09-11T05:10:00.405620+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:10:00.405620+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1511}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12911,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307526+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:57.307526+0000","last_peered":"2024-09-11T05:09:57.307526+0000","last_clean":"2024-09-11T05:09:57.307526+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:57.307526+0000","last_undegraded":"2024-09-11T05:09:57.307526+0000","last_fullsized":"2024-09-11T05:09:57.307526+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16757,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308709+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:57.308709+0000","last_peered":"2024-09-11T05:09:57.308709+0000","last_clean":"2024-09-11T05:09:57.308709+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:57.308709+0000","last_undegraded":"2024-09-11T05:09:57.308709+0000","last_fullsized":"2024-09-11T05:09:57.308709+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13979,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306716+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:57.306716+0000","last_peered":"2024-09-11T05:09:57.306716+0000","last_clean":"2024-09-11T05:09:57.306716+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:57.306716+0000","last_undegraded":"2024-09-11T05:09:57.306716+0000","last_fullsized":"2024-09-11T05:09:57.306716+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17167,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308998+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:57.308998+0000","last_peered":"2024-09-11T05:09:57.308998+0000","last_clean":"2024-09-11T05:09:57.308998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:57.308998+0000","last_undegraded":"2024-09-11T05:09:57.308998+0000","last_fullsized":"2024-09-11T05:09:57.308998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17394,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.309470+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:57.309470+0000","last_peered":"2024-09-11T05:09:57.309470+0000","last_clean":"2024-09-11T05:09:57.309470+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:57.309470+0000","last_undegraded":"2024-09-11T05:09:57.309470+0000","last_fullsized":"2024-09-11T05:09:57.309470+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17509,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311539+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311539+0000","last_peered":"2024-09-11T05:09:57.311539+0000","last_clean":"2024-09-11T05:09:57.311539+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311539+0000","last_undegraded":"2024-09-11T05:09:57.311539+0000","last_fullsized":"2024-09-11T05:09:57.311539+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17731,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307005+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:57.307005+0000","last_peered":"2024-09-11T05:09:57.307005+0000","last_clean":"2024-09-11T05:09:57.307005+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:57.307005+0000","last_undegraded":"2024-09-11T05:09:57.307005+0000","last_fullsized":"2024-09-11T05:09:57.307005+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17713,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.409213+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:09:57.409213+0000","last_peered":"2024-09-11T05:09:57.409213+0000","last_clean":"2024-09-11T05:09:57.409213+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:09:57.409213+0000","last_undegraded":"2024-09-11T05:09:57.409213+0000","last_fullsized":"2024-09-11T05:09:57.409213+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17323,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308493+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:57.308493+0000","last_peered":"2024-09-11T05:09:57.308493+0000","last_clean":"2024-09-11T05:09:57.308493+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:57.308493+0000","last_undegraded":"2024-09-11T05:09:57.308493+0000","last_fullsized":"2024-09-11T05:09:57.308493+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12817,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:57.309063+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:57.309063+0000","last_peered":"2024-09-11T05:09:57.309063+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:57.309063+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17035,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307639+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:57.307639+0000","last_peered":"2024-09-11T05:09:57.307639+0000","last_clean":"2024-09-11T05:09:57.307639+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:57.307639+0000","last_undegraded":"2024-09-11T05:09:57.307639+0000","last_fullsized":"2024-09-11T05:09:57.307639+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17549,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307146+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:57.307146+0000","last_peered":"2024-09-11T05:09:57.307146+0000","last_clean":"2024-09-11T05:09:57.307146+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:57.307146+0000","last_undegraded":"2024-09-11T05:09:57.307146+0000","last_fullsized":"2024-09-11T05:09:57.307146+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12745,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306929+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:57.306929+0000","last_peered":"2024-09-11T05:09:57.306929+0000","last_clean":"2024-09-11T05:09:57.306929+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:57.306929+0000","last_undegraded":"2024-09-11T05:09:57.306929+0000","last_fullsized":"2024-09-11T05:09:57.306929+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307165+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:57.307165+0000","last_peered":"2024-09-11T05:09:57.307165+0000","last_clean":"2024-09-11T05:09:57.307165+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:57.307165+0000","last_undegraded":"2024-09-11T05:09:57.307165+0000","last_fullsized":"2024-09-11T05:09:57.307165+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307037+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:57.307037+0000","last_peered":"2024-09-11T05:09:57.307037+0000","last_clean":"2024-09-11T05:09:57.307037+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:57.307037+0000","last_undegraded":"2024-09-11T05:09:57.307037+0000","last_fullsized":"2024-09-11T05:09:57.307037+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744380,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627460,"statfs":{"total":96636764160,"available":95874519040,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721660,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650180,"statfs":{"total":96636764160,"available":95897784320,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636032,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735808,"statfs":{"total":96636764160,"available":95985467392,"internally_reserved":0,"allocated":507904,"data_stored":202134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:08.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:08.557 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:08.569 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-11T05:10:07.953635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815860,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671500,"statfs":{"total":386547056640,"available":383663616000,"internally_reserved":0,"allocated":1994752,"data_stored":771672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12781,"num_write_kb":-51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-624,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001627"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":16100,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:05.406757+0000","last_change":"2024-09-11T05:09:55.662190+0000","last_active":"2024-09-11T05:10:05.406757+0000","last_peered":"2024-09-11T05:10:05.406757+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:10:05.406757+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12911,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307526+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:57.307526+0000","last_peered":"2024-09-11T05:09:57.307526+0000","last_clean":"2024-09-11T05:09:57.307526+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:57.307526+0000","last_undegraded":"2024-09-11T05:09:57.307526+0000","last_fullsized":"2024-09-11T05:09:57.307526+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16757,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308709+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:57.308709+0000","last_peered":"2024-09-11T05:09:57.308709+0000","last_clean":"2024-09-11T05:09:57.308709+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:57.308709+0000","last_undegraded":"2024-09-11T05:09:57.308709+0000","last_fullsized":"2024-09-11T05:09:57.308709+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13979,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306716+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:57.306716+0000","last_peered":"2024-09-11T05:09:57.306716+0000","last_clean":"2024-09-11T05:09:57.306716+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:57.306716+0000","last_undegraded":"2024-09-11T05:09:57.306716+0000","last_fullsized":"2024-09-11T05:09:57.306716+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17167,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308998+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:57.308998+0000","last_peered":"2024-09-11T05:09:57.308998+0000","last_clean":"2024-09-11T05:09:57.308998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:57.308998+0000","last_undegraded":"2024-09-11T05:09:57.308998+0000","last_fullsized":"2024-09-11T05:09:57.308998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17394,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.309470+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:57.309470+0000","last_peered":"2024-09-11T05:09:57.309470+0000","last_clean":"2024-09-11T05:09:57.309470+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:57.309470+0000","last_undegraded":"2024-09-11T05:09:57.309470+0000","last_fullsized":"2024-09-11T05:09:57.309470+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17509,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311539+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311539+0000","last_peered":"2024-09-11T05:09:57.311539+0000","last_clean":"2024-09-11T05:09:57.311539+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311539+0000","last_undegraded":"2024-09-11T05:09:57.311539+0000","last_fullsized":"2024-09-11T05:09:57.311539+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17731,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307005+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:57.307005+0000","last_peered":"2024-09-11T05:09:57.307005+0000","last_clean":"2024-09-11T05:09:57.307005+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:57.307005+0000","last_undegraded":"2024-09-11T05:09:57.307005+0000","last_fullsized":"2024-09-11T05:09:57.307005+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17713,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.409213+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:09:57.409213+0000","last_peered":"2024-09-11T05:09:57.409213+0000","last_clean":"2024-09-11T05:09:57.409213+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:09:57.409213+0000","last_undegraded":"2024-09-11T05:09:57.409213+0000","last_fullsized":"2024-09-11T05:09:57.409213+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17323,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308493+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:57.308493+0000","last_peered":"2024-09-11T05:09:57.308493+0000","last_clean":"2024-09-11T05:09:57.308493+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:57.308493+0000","last_undegraded":"2024-09-11T05:09:57.308493+0000","last_fullsized":"2024-09-11T05:09:57.308493+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12817,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:57.309063+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:57.309063+0000","last_peered":"2024-09-11T05:09:57.309063+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:57.309063+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17035,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307639+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:57.307639+0000","last_peered":"2024-09-11T05:09:57.307639+0000","last_clean":"2024-09-11T05:09:57.307639+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:57.307639+0000","last_undegraded":"2024-09-11T05:09:57.307639+0000","last_fullsized":"2024-09-11T05:09:57.307639+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17549,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307146+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:57.307146+0000","last_peered":"2024-09-11T05:09:57.307146+0000","last_clean":"2024-09-11T05:09:57.307146+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:57.307146+0000","last_undegraded":"2024-09-11T05:09:57.307146+0000","last_fullsized":"2024-09-11T05:09:57.307146+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12745,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306929+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:57.306929+0000","last_peered":"2024-09-11T05:09:57.306929+0000","last_clean":"2024-09-11T05:09:57.306929+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:57.306929+0000","last_undegraded":"2024-09-11T05:09:57.306929+0000","last_fullsized":"2024-09-11T05:09:57.306929+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307165+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:57.307165+0000","last_peered":"2024-09-11T05:09:57.307165+0000","last_clean":"2024-09-11T05:09:57.307165+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:57.307165+0000","last_undegraded":"2024-09-11T05:09:57.307165+0000","last_fullsized":"2024-09-11T05:09:57.307165+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307037+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:57.307037+0000","last_peered":"2024-09-11T05:09:57.307037+0000","last_clean":"2024-09-11T05:09:57.307037+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:57.307037+0000","last_undegraded":"2024-09-11T05:09:57.307037+0000","last_fullsized":"2024-09-11T05:09:57.307037+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744380,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627460,"statfs":{"total":96636764160,"available":95874519040,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721660,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650180,"statfs":{"total":96636764160,"available":95897784320,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627712,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744128,"statfs":{"total":96636764160,"available":95993987072,"internally_reserved":0,"allocated":507904,"data_stored":202134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:08.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:08.959 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:08.971 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-11T05:10:08.926835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815860,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671500,"statfs":{"total":386547056640,"available":383663616000,"internally_reserved":0,"allocated":1994752,"data_stored":771672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12781,"num_write_kb":-51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-624,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.974511"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":16100,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:05.406757+0000","last_change":"2024-09-11T05:09:55.662190+0000","last_active":"2024-09-11T05:10:05.406757+0000","last_peered":"2024-09-11T05:10:05.406757+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:10:05.406757+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12911,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307526+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:57.307526+0000","last_peered":"2024-09-11T05:09:57.307526+0000","last_clean":"2024-09-11T05:09:57.307526+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:57.307526+0000","last_undegraded":"2024-09-11T05:09:57.307526+0000","last_fullsized":"2024-09-11T05:09:57.307526+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16757,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308709+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:57.308709+0000","last_peered":"2024-09-11T05:09:57.308709+0000","last_clean":"2024-09-11T05:09:57.308709+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:57.308709+0000","last_undegraded":"2024-09-11T05:09:57.308709+0000","last_fullsized":"2024-09-11T05:09:57.308709+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13979,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306716+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:57.306716+0000","last_peered":"2024-09-11T05:09:57.306716+0000","last_clean":"2024-09-11T05:09:57.306716+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:57.306716+0000","last_undegraded":"2024-09-11T05:09:57.306716+0000","last_fullsized":"2024-09-11T05:09:57.306716+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17167,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308998+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:57.308998+0000","last_peered":"2024-09-11T05:09:57.308998+0000","last_clean":"2024-09-11T05:09:57.308998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:57.308998+0000","last_undegraded":"2024-09-11T05:09:57.308998+0000","last_fullsized":"2024-09-11T05:09:57.308998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17394,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.309470+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:57.309470+0000","last_peered":"2024-09-11T05:09:57.309470+0000","last_clean":"2024-09-11T05:09:57.309470+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:57.309470+0000","last_undegraded":"2024-09-11T05:09:57.309470+0000","last_fullsized":"2024-09-11T05:09:57.309470+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17509,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311539+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311539+0000","last_peered":"2024-09-11T05:09:57.311539+0000","last_clean":"2024-09-11T05:09:57.311539+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311539+0000","last_undegraded":"2024-09-11T05:09:57.311539+0000","last_fullsized":"2024-09-11T05:09:57.311539+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17731,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307005+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:57.307005+0000","last_peered":"2024-09-11T05:09:57.307005+0000","last_clean":"2024-09-11T05:09:57.307005+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:57.307005+0000","last_undegraded":"2024-09-11T05:09:57.307005+0000","last_fullsized":"2024-09-11T05:09:57.307005+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17713,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.409213+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:09:57.409213+0000","last_peered":"2024-09-11T05:09:57.409213+0000","last_clean":"2024-09-11T05:09:57.409213+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:09:57.409213+0000","last_undegraded":"2024-09-11T05:09:57.409213+0000","last_fullsized":"2024-09-11T05:09:57.409213+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17323,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308493+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:57.308493+0000","last_peered":"2024-09-11T05:09:57.308493+0000","last_clean":"2024-09-11T05:09:57.308493+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:57.308493+0000","last_undegraded":"2024-09-11T05:09:57.308493+0000","last_fullsized":"2024-09-11T05:09:57.308493+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12817,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:57.309063+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:57.309063+0000","last_peered":"2024-09-11T05:09:57.309063+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:57.309063+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17035,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307639+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:57.307639+0000","last_peered":"2024-09-11T05:09:57.307639+0000","last_clean":"2024-09-11T05:09:57.307639+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:57.307639+0000","last_undegraded":"2024-09-11T05:09:57.307639+0000","last_fullsized":"2024-09-11T05:09:57.307639+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17549,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307146+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:57.307146+0000","last_peered":"2024-09-11T05:09:57.307146+0000","last_clean":"2024-09-11T05:09:57.307146+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:57.307146+0000","last_undegraded":"2024-09-11T05:09:57.307146+0000","last_fullsized":"2024-09-11T05:09:57.307146+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12745,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306929+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:57.306929+0000","last_peered":"2024-09-11T05:09:57.306929+0000","last_clean":"2024-09-11T05:09:57.306929+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:57.306929+0000","last_undegraded":"2024-09-11T05:09:57.306929+0000","last_fullsized":"2024-09-11T05:09:57.306929+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307165+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:57.307165+0000","last_peered":"2024-09-11T05:09:57.307165+0000","last_clean":"2024-09-11T05:09:57.307165+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:57.307165+0000","last_undegraded":"2024-09-11T05:09:57.307165+0000","last_fullsized":"2024-09-11T05:09:57.307165+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307037+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:57.307037+0000","last_peered":"2024-09-11T05:09:57.307037+0000","last_clean":"2024-09-11T05:09:57.307037+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:57.307037+0000","last_undegraded":"2024-09-11T05:09:57.307037+0000","last_fullsized":"2024-09-11T05:09:57.307037+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744380,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627460,"statfs":{"total":96636764160,"available":95874519040,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721660,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650180,"statfs":{"total":96636764160,"available":95897784320,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627712,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744128,"statfs":{"total":96636764160,"available":95993987072,"internally_reserved":0,"allocated":507904,"data_stored":202134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:08.972 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-11T05:10:08.973 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12514", 'reported_seq': 16100, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:10:05.406757+0000', 'last_change': '2024-09-11T05:09:55.662190+0000', 'last_active': '2024-09-11T05:10:05.406757+0000', 'last_peered': '2024-09-11T05:10:05.406757+0000', 'last_clean': '2024-09-11T04:54:56.683804+0000', 'last_became_active': '2024-09-11T04:56:20.611034+0000', 'last_became_peered': '2024-09-11T04:56:20.611034+0000', 'last_unstale': '2024-09-11T05:10:05.406757+0000', 'last_undegraded': '2024-09-11T04:56:20.317188+0000', 'last_fullsized': '2024-09-11T04:56:20.273688+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 571, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12514, 'num_write_kb': 50056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1741, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -571}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:10:08.973 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:10:08.973 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 12817, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:09:57.309063+0000', 'last_change': '2024-09-11T04:56:20.671613+0000', 'last_active': '2024-09-11T05:09:57.309063+0000', 'last_peered': '2024-09-11T05:09:57.309063+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:09:57.309063+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:10:08.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:10:09.430 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:09.430 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2872/16 objects degraded (17950.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":936,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":26944,"bytes_used":2883440640,"bytes_avail":383663616000,"bytes_total":386547056640,"degraded_objects":2872,"degraded_total":16,"degraded_ratio":179.5},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:10:09.446 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-11T05:10:09.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:09.789 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:09.802 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-11T05:10:08.926835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815860,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671500,"statfs":{"total":386547056640,"available":383663616000,"internally_reserved":0,"allocated":1994752,"data_stored":771672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12781,"num_write_kb":-51137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-624,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.974511"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":16100,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:05.406757+0000","last_change":"2024-09-11T05:09:55.662190+0000","last_active":"2024-09-11T05:10:05.406757+0000","last_peered":"2024-09-11T05:10:05.406757+0000","last_clean":"2024-09-11T04:54:56.683804+0000","last_became_active":"2024-09-11T04:56:20.611034+0000","last_became_peered":"2024-09-11T04:56:20.611034+0000","last_unstale":"2024-09-11T05:10:05.406757+0000","last_undegraded":"2024-09-11T04:56:20.317188+0000","last_fullsized":"2024-09-11T04:56:20.273688+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12911,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307526+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:57.307526+0000","last_peered":"2024-09-11T05:09:57.307526+0000","last_clean":"2024-09-11T05:09:57.307526+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:57.307526+0000","last_undegraded":"2024-09-11T05:09:57.307526+0000","last_fullsized":"2024-09-11T05:09:57.307526+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16757,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308709+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:09:57.308709+0000","last_peered":"2024-09-11T05:09:57.308709+0000","last_clean":"2024-09-11T05:09:57.308709+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:09:57.308709+0000","last_undegraded":"2024-09-11T05:09:57.308709+0000","last_fullsized":"2024-09-11T05:09:57.308709+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13979,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306716+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:57.306716+0000","last_peered":"2024-09-11T05:09:57.306716+0000","last_clean":"2024-09-11T05:09:57.306716+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:57.306716+0000","last_undegraded":"2024-09-11T05:09:57.306716+0000","last_fullsized":"2024-09-11T05:09:57.306716+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17167,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308998+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:09:57.308998+0000","last_peered":"2024-09-11T05:09:57.308998+0000","last_clean":"2024-09-11T05:09:57.308998+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:09:57.308998+0000","last_undegraded":"2024-09-11T05:09:57.308998+0000","last_fullsized":"2024-09-11T05:09:57.308998+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17394,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.309470+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:09:57.309470+0000","last_peered":"2024-09-11T05:09:57.309470+0000","last_clean":"2024-09-11T05:09:57.309470+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:09:57.309470+0000","last_undegraded":"2024-09-11T05:09:57.309470+0000","last_fullsized":"2024-09-11T05:09:57.309470+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17509,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311539+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311539+0000","last_peered":"2024-09-11T05:09:57.311539+0000","last_clean":"2024-09-11T05:09:57.311539+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311539+0000","last_undegraded":"2024-09-11T05:09:57.311539+0000","last_fullsized":"2024-09-11T05:09:57.311539+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17731,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307005+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:57.307005+0000","last_peered":"2024-09-11T05:09:57.307005+0000","last_clean":"2024-09-11T05:09:57.307005+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:57.307005+0000","last_undegraded":"2024-09-11T05:09:57.307005+0000","last_fullsized":"2024-09-11T05:09:57.307005+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17713,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.409213+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:09:57.409213+0000","last_peered":"2024-09-11T05:09:57.409213+0000","last_clean":"2024-09-11T05:09:57.409213+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:09:57.409213+0000","last_undegraded":"2024-09-11T05:09:57.409213+0000","last_fullsized":"2024-09-11T05:09:57.409213+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17323,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.308493+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:09:57.308493+0000","last_peered":"2024-09-11T05:09:57.308493+0000","last_clean":"2024-09-11T05:09:57.308493+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:09:57.308493+0000","last_undegraded":"2024-09-11T05:09:57.308493+0000","last_fullsized":"2024-09-11T05:09:57.308493+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":12817,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-11T05:09:57.309063+0000","last_change":"2024-09-11T04:56:20.671613+0000","last_active":"2024-09-11T05:09:57.309063+0000","last_peered":"2024-09-11T05:09:57.309063+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:09:57.309063+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17035,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307639+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:57.307639+0000","last_peered":"2024-09-11T05:09:57.307639+0000","last_clean":"2024-09-11T05:09:57.307639+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:57.307639+0000","last_undegraded":"2024-09-11T05:09:57.307639+0000","last_fullsized":"2024-09-11T05:09:57.307639+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17549,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307146+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:57.307146+0000","last_peered":"2024-09-11T05:09:57.307146+0000","last_clean":"2024-09-11T05:09:57.307146+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:57.307146+0000","last_undegraded":"2024-09-11T05:09:57.307146+0000","last_fullsized":"2024-09-11T05:09:57.307146+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12745,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306929+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:57.306929+0000","last_peered":"2024-09-11T05:09:57.306929+0000","last_clean":"2024-09-11T05:09:57.306929+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:57.306929+0000","last_undegraded":"2024-09-11T05:09:57.306929+0000","last_fullsized":"2024-09-11T05:09:57.306929+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307165+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:57.307165+0000","last_peered":"2024-09-11T05:09:57.307165+0000","last_clean":"2024-09-11T05:09:57.307165+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:57.307165+0000","last_undegraded":"2024-09-11T05:09:57.307165+0000","last_fullsized":"2024-09-11T05:09:57.307165+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307037+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:57.307037+0000","last_peered":"2024-09-11T05:09:57.307037+0000","last_clean":"2024-09-11T05:09:57.307037+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:57.307037+0000","last_undegraded":"2024-09-11T05:09:57.307037+0000","last_fullsized":"2024-09-11T05:09:57.307037+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744380,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627460,"statfs":{"total":96636764160,"available":95874519040,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721660,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650180,"statfs":{"total":96636764160,"available":95897784320,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627712,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744128,"statfs":{"total":96636764160,"available":95993987072,"internally_reserved":0,"allocated":507904,"data_stored":202134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:12.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:13.121 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:13.122 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:13.134 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-11T05:10:11.954255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815892,"kb_used_data":1980,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671468,"statfs":{"total":386547056640,"available":383663583232,"internally_reserved":0,"allocated":2027520,"data_stored":784050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2027,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001224"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17255,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:09.011604+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:09.011604+0000","last_peered":"2024-09-11T05:10:09.011604+0000","last_clean":"2024-09-11T05:10:09.011604+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:09.011604+0000","last_undegraded":"2024-09-11T05:10:09.011604+0000","last_fullsized":"2024-09-11T05:10:09.011604+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12911,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307526+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:57.307526+0000","last_peered":"2024-09-11T05:09:57.307526+0000","last_clean":"2024-09-11T05:09:57.307526+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:57.307526+0000","last_undegraded":"2024-09-11T05:09:57.307526+0000","last_fullsized":"2024-09-11T05:09:57.307526+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13979,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306716+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:57.306716+0000","last_peered":"2024-09-11T05:09:57.306716+0000","last_clean":"2024-09-11T05:09:57.306716+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:57.306716+0000","last_undegraded":"2024-09-11T05:09:57.306716+0000","last_fullsized":"2024-09-11T05:09:57.306716+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17171,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929358+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:08.929358+0000","last_peered":"2024-09-11T05:10:08.929358+0000","last_clean":"2024-09-11T05:10:08.929358+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:08.929358+0000","last_undegraded":"2024-09-11T05:10:08.929358+0000","last_fullsized":"2024-09-11T05:10:08.929358+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17398,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930137+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:08.930137+0000","last_peered":"2024-09-11T05:10:08.930137+0000","last_clean":"2024-09-11T05:10:08.930137+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:08.930137+0000","last_undegraded":"2024-09-11T05:10:08.930137+0000","last_fullsized":"2024-09-11T05:10:08.930137+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17509,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311539+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311539+0000","last_peered":"2024-09-11T05:09:57.311539+0000","last_clean":"2024-09-11T05:09:57.311539+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311539+0000","last_undegraded":"2024-09-11T05:09:57.311539+0000","last_fullsized":"2024-09-11T05:09:57.311539+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17731,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307005+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:57.307005+0000","last_peered":"2024-09-11T05:09:57.307005+0000","last_clean":"2024-09-11T05:09:57.307005+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:57.307005+0000","last_undegraded":"2024-09-11T05:09:57.307005+0000","last_fullsized":"2024-09-11T05:09:57.307005+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17717,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930089+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:08.930089+0000","last_peered":"2024-09-11T05:10:08.930089+0000","last_clean":"2024-09-11T05:10:08.930089+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:08.930089+0000","last_undegraded":"2024-09-11T05:10:08.930089+0000","last_fullsized":"2024-09-11T05:10:08.930089+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":13854,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:10.405792+0000","last_change":"2024-09-11T05:10:07.575949+0000","last_active":"2024-09-11T05:10:10.405792+0000","last_peered":"2024-09-11T05:10:10.405792+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:10.405792+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17035,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307639+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:57.307639+0000","last_peered":"2024-09-11T05:09:57.307639+0000","last_clean":"2024-09-11T05:09:57.307639+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:57.307639+0000","last_undegraded":"2024-09-11T05:09:57.307639+0000","last_fullsized":"2024-09-11T05:09:57.307639+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17549,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307146+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:57.307146+0000","last_peered":"2024-09-11T05:09:57.307146+0000","last_clean":"2024-09-11T05:09:57.307146+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:57.307146+0000","last_undegraded":"2024-09-11T05:09:57.307146+0000","last_fullsized":"2024-09-11T05:09:57.307146+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12745,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306929+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:57.306929+0000","last_peered":"2024-09-11T05:09:57.306929+0000","last_clean":"2024-09-11T05:09:57.306929+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:57.306929+0000","last_undegraded":"2024-09-11T05:09:57.306929+0000","last_fullsized":"2024-09-11T05:09:57.306929+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307165+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:57.307165+0000","last_peered":"2024-09-11T05:09:57.307165+0000","last_clean":"2024-09-11T05:09:57.307165+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:57.307165+0000","last_undegraded":"2024-09-11T05:09:57.307165+0000","last_fullsized":"2024-09-11T05:09:57.307165+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307037+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:57.307037+0000","last_peered":"2024-09-11T05:09:57.307037+0000","last_clean":"2024-09-11T05:09:57.307037+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:57.307037+0000","last_undegraded":"2024-09-11T05:09:57.307037+0000","last_fullsized":"2024-09-11T05:09:57.307037+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627712,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744128,"statfs":{"total":96636764160,"available":95993987072,"internally_reserved":0,"allocated":507904,"data_stored":202134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:13.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:13.470 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:13.470 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:13.483 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-11T05:10:11.954255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815892,"kb_used_data":1980,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671468,"statfs":{"total":386547056640,"available":383663583232,"internally_reserved":0,"allocated":2027520,"data_stored":784050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2027,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001224"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17255,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:09.011604+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:09.011604+0000","last_peered":"2024-09-11T05:10:09.011604+0000","last_clean":"2024-09-11T05:10:09.011604+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:09.011604+0000","last_undegraded":"2024-09-11T05:10:09.011604+0000","last_fullsized":"2024-09-11T05:10:09.011604+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12911,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307526+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:09:57.307526+0000","last_peered":"2024-09-11T05:09:57.307526+0000","last_clean":"2024-09-11T05:09:57.307526+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:09:57.307526+0000","last_undegraded":"2024-09-11T05:09:57.307526+0000","last_fullsized":"2024-09-11T05:09:57.307526+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13979,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306716+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:09:57.306716+0000","last_peered":"2024-09-11T05:09:57.306716+0000","last_clean":"2024-09-11T05:09:57.306716+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:09:57.306716+0000","last_undegraded":"2024-09-11T05:09:57.306716+0000","last_fullsized":"2024-09-11T05:09:57.306716+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17171,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929358+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:08.929358+0000","last_peered":"2024-09-11T05:10:08.929358+0000","last_clean":"2024-09-11T05:10:08.929358+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:08.929358+0000","last_undegraded":"2024-09-11T05:10:08.929358+0000","last_fullsized":"2024-09-11T05:10:08.929358+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17398,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930137+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:08.930137+0000","last_peered":"2024-09-11T05:10:08.930137+0000","last_clean":"2024-09-11T05:10:08.930137+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:08.930137+0000","last_undegraded":"2024-09-11T05:10:08.930137+0000","last_fullsized":"2024-09-11T05:10:08.930137+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17509,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.311539+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:09:57.311539+0000","last_peered":"2024-09-11T05:09:57.311539+0000","last_clean":"2024-09-11T05:09:57.311539+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:09:57.311539+0000","last_undegraded":"2024-09-11T05:09:57.311539+0000","last_fullsized":"2024-09-11T05:09:57.311539+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17731,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307005+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:09:57.307005+0000","last_peered":"2024-09-11T05:09:57.307005+0000","last_clean":"2024-09-11T05:09:57.307005+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:09:57.307005+0000","last_undegraded":"2024-09-11T05:09:57.307005+0000","last_fullsized":"2024-09-11T05:09:57.307005+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17717,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930089+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:08.930089+0000","last_peered":"2024-09-11T05:10:08.930089+0000","last_clean":"2024-09-11T05:10:08.930089+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:08.930089+0000","last_undegraded":"2024-09-11T05:10:08.930089+0000","last_fullsized":"2024-09-11T05:10:08.930089+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":13854,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:10.405792+0000","last_change":"2024-09-11T05:10:07.575949+0000","last_active":"2024-09-11T05:10:10.405792+0000","last_peered":"2024-09-11T05:10:10.405792+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:10.405792+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17035,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307639+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:09:57.307639+0000","last_peered":"2024-09-11T05:09:57.307639+0000","last_clean":"2024-09-11T05:09:57.307639+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:09:57.307639+0000","last_undegraded":"2024-09-11T05:09:57.307639+0000","last_fullsized":"2024-09-11T05:09:57.307639+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17549,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307146+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:09:57.307146+0000","last_peered":"2024-09-11T05:09:57.307146+0000","last_clean":"2024-09-11T05:09:57.307146+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:09:57.307146+0000","last_undegraded":"2024-09-11T05:09:57.307146+0000","last_fullsized":"2024-09-11T05:09:57.307146+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12745,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.306929+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:09:57.306929+0000","last_peered":"2024-09-11T05:09:57.306929+0000","last_clean":"2024-09-11T05:09:57.306929+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:09:57.306929+0000","last_undegraded":"2024-09-11T05:09:57.306929+0000","last_fullsized":"2024-09-11T05:09:57.306929+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12558,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307165+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:09:57.307165+0000","last_peered":"2024-09-11T05:09:57.307165+0000","last_clean":"2024-09-11T05:09:57.307165+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:09:57.307165+0000","last_undegraded":"2024-09-11T05:09:57.307165+0000","last_fullsized":"2024-09-11T05:09:57.307165+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T05:09:57.307037+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:09:57.307037+0000","last_peered":"2024-09-11T05:09:57.307037+0000","last_clean":"2024-09-11T05:09:57.307037+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:09:57.307037+0000","last_undegraded":"2024-09-11T05:09:57.307037+0000","last_fullsized":"2024-09-11T05:09:57.307037+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627712,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744128,"statfs":{"total":96636764160,"available":95993987072,"internally_reserved":0,"allocated":507904,"data_stored":202134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:13.484 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:10:13.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 13854, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:10:10.405792+0000', 'last_change': '2024-09-11T05:10:07.575949+0000', 'last_active': '2024-09-11T05:10:10.405792+0000', 'last_peered': '2024-09-11T05:10:10.405792+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:10:10.405792+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1785, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 544, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1782}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:10:13.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:10:13.928 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:13.928 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1785/16 objects degraded (11156.250%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":940,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":26944,"bytes_used":2883473408,"bytes_avail":383663583232,"bytes_total":386547056640,"degraded_objects":1785,"degraded_total":16,"degraded_ratio":111.5625,"recovering_objects_per_sec":253,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2027,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:10:13.942 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:10:13.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:14.249 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:14.250 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:14.262 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-11T05:10:13.954597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824236,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663124,"statfs":{"total":386547056640,"available":383655038976,"internally_reserved":0,"allocated":2048000,"data_stored":784140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001278"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17255,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:09.011604+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:09.011604+0000","last_peered":"2024-09-11T05:10:09.011604+0000","last_clean":"2024-09-11T05:10:09.011604+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:09.011604+0000","last_undegraded":"2024-09-11T05:10:09.011604+0000","last_fullsized":"2024-09-11T05:10:09.011604+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12915,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.934143+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:08.934143+0000","last_peered":"2024-09-11T05:10:08.934143+0000","last_clean":"2024-09-11T05:10:08.934143+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:08.934143+0000","last_undegraded":"2024-09-11T05:10:08.934143+0000","last_fullsized":"2024-09-11T05:10:08.934143+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13983,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932967+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:08.932967+0000","last_peered":"2024-09-11T05:10:08.932967+0000","last_clean":"2024-09-11T05:10:08.932967+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:08.932967+0000","last_undegraded":"2024-09-11T05:10:08.932967+0000","last_fullsized":"2024-09-11T05:10:08.932967+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17171,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929358+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:08.929358+0000","last_peered":"2024-09-11T05:10:08.929358+0000","last_clean":"2024-09-11T05:10:08.929358+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:08.929358+0000","last_undegraded":"2024-09-11T05:10:08.929358+0000","last_fullsized":"2024-09-11T05:10:08.929358+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17398,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930137+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:08.930137+0000","last_peered":"2024-09-11T05:10:08.930137+0000","last_clean":"2024-09-11T05:10:08.930137+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:08.930137+0000","last_undegraded":"2024-09-11T05:10:08.930137+0000","last_fullsized":"2024-09-11T05:10:08.930137+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17513,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933634+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:08.933634+0000","last_peered":"2024-09-11T05:10:08.933634+0000","last_clean":"2024-09-11T05:10:08.933634+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:08.933634+0000","last_undegraded":"2024-09-11T05:10:08.933634+0000","last_fullsized":"2024-09-11T05:10:08.933634+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17735,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933151+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:08.933151+0000","last_peered":"2024-09-11T05:10:08.933151+0000","last_clean":"2024-09-11T05:10:08.933151+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:08.933151+0000","last_undegraded":"2024-09-11T05:10:08.933151+0000","last_fullsized":"2024-09-11T05:10:08.933151+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17717,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930089+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:08.930089+0000","last_peered":"2024-09-11T05:10:08.930089+0000","last_clean":"2024-09-11T05:10:08.930089+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:08.930089+0000","last_undegraded":"2024-09-11T05:10:08.930089+0000","last_fullsized":"2024-09-11T05:10:08.930089+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":13854,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:10.405792+0000","last_change":"2024-09-11T05:10:07.575949+0000","last_active":"2024-09-11T05:10:10.405792+0000","last_peered":"2024-09-11T05:10:10.405792+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:10.405792+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17039,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:08.933330+0000","last_peered":"2024-09-11T05:10:08.933330+0000","last_clean":"2024-09-11T05:10:08.933330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:08.933330+0000","last_undegraded":"2024-09-11T05:10:08.933330+0000","last_fullsized":"2024-09-11T05:10:08.933330+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17553,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932973+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:08.932973+0000","last_peered":"2024-09-11T05:10:08.932973+0000","last_clean":"2024-09-11T05:10:08.932973+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:08.932973+0000","last_undegraded":"2024-09-11T05:10:08.932973+0000","last_fullsized":"2024-09-11T05:10:08.932973+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12749,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933058+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:08.933058+0000","last_peered":"2024-09-11T05:10:08.933058+0000","last_clean":"2024-09-11T05:10:08.933058+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:08.933058+0000","last_undegraded":"2024-09-11T05:10:08.933058+0000","last_fullsized":"2024-09-11T05:10:08.933058+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12562,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933136+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:08.933136+0000","last_peered":"2024-09-11T05:10:08.933136+0000","last_clean":"2024-09-11T05:10:08.933136+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:08.933136+0000","last_undegraded":"2024-09-11T05:10:08.933136+0000","last_fullsized":"2024-09-11T05:10:08.933136+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933256+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:08.933256+0000","last_peered":"2024-09-11T05:10:08.933256+0000","last_clean":"2024-09-11T05:10:08.933256+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:08.933256+0000","last_undegraded":"2024-09-11T05:10:08.933256+0000","last_fullsized":"2024-09-11T05:10:08.933256+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636040,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735800,"statfs":{"total":96636764160,"available":95985459200,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:17.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:17.590 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:17.603 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-11T05:10:15.954933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824236,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663124,"statfs":{"total":386547056640,"available":383655038976,"internally_reserved":0,"allocated":2048000,"data_stored":784140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.035967"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17255,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:09.011604+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:09.011604+0000","last_peered":"2024-09-11T05:10:09.011604+0000","last_clean":"2024-09-11T05:10:09.011604+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:09.011604+0000","last_undegraded":"2024-09-11T05:10:09.011604+0000","last_fullsized":"2024-09-11T05:10:09.011604+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12915,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.934143+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:08.934143+0000","last_peered":"2024-09-11T05:10:08.934143+0000","last_clean":"2024-09-11T05:10:08.934143+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:08.934143+0000","last_undegraded":"2024-09-11T05:10:08.934143+0000","last_fullsized":"2024-09-11T05:10:08.934143+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13983,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932967+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:08.932967+0000","last_peered":"2024-09-11T05:10:08.932967+0000","last_clean":"2024-09-11T05:10:08.932967+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:08.932967+0000","last_undegraded":"2024-09-11T05:10:08.932967+0000","last_fullsized":"2024-09-11T05:10:08.932967+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17171,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929358+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:08.929358+0000","last_peered":"2024-09-11T05:10:08.929358+0000","last_clean":"2024-09-11T05:10:08.929358+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:08.929358+0000","last_undegraded":"2024-09-11T05:10:08.929358+0000","last_fullsized":"2024-09-11T05:10:08.929358+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17398,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930137+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:08.930137+0000","last_peered":"2024-09-11T05:10:08.930137+0000","last_clean":"2024-09-11T05:10:08.930137+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:08.930137+0000","last_undegraded":"2024-09-11T05:10:08.930137+0000","last_fullsized":"2024-09-11T05:10:08.930137+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17513,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933634+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:08.933634+0000","last_peered":"2024-09-11T05:10:08.933634+0000","last_clean":"2024-09-11T05:10:08.933634+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:08.933634+0000","last_undegraded":"2024-09-11T05:10:08.933634+0000","last_fullsized":"2024-09-11T05:10:08.933634+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17735,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933151+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:08.933151+0000","last_peered":"2024-09-11T05:10:08.933151+0000","last_clean":"2024-09-11T05:10:08.933151+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:08.933151+0000","last_undegraded":"2024-09-11T05:10:08.933151+0000","last_fullsized":"2024-09-11T05:10:08.933151+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17718,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930140+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:08.930140+0000","last_peered":"2024-09-11T05:10:08.930140+0000","last_clean":"2024-09-11T05:10:08.930140+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:08.930140+0000","last_undegraded":"2024-09-11T05:10:08.930140+0000","last_fullsized":"2024-09-11T05:10:08.930140+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":15584,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:15.409547+0000","last_change":"2024-09-11T05:10:07.575949+0000","last_active":"2024-09-11T05:10:15.409547+0000","last_peered":"2024-09-11T05:10:15.409547+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:15.409547+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17039,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:08.933330+0000","last_peered":"2024-09-11T05:10:08.933330+0000","last_clean":"2024-09-11T05:10:08.933330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:08.933330+0000","last_undegraded":"2024-09-11T05:10:08.933330+0000","last_fullsized":"2024-09-11T05:10:08.933330+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17553,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932973+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:08.932973+0000","last_peered":"2024-09-11T05:10:08.932973+0000","last_clean":"2024-09-11T05:10:08.932973+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:08.932973+0000","last_undegraded":"2024-09-11T05:10:08.932973+0000","last_fullsized":"2024-09-11T05:10:08.932973+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12749,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933058+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:08.933058+0000","last_peered":"2024-09-11T05:10:08.933058+0000","last_clean":"2024-09-11T05:10:08.933058+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:08.933058+0000","last_undegraded":"2024-09-11T05:10:08.933058+0000","last_fullsized":"2024-09-11T05:10:08.933058+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12562,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933136+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:08.933136+0000","last_peered":"2024-09-11T05:10:08.933136+0000","last_clean":"2024-09-11T05:10:08.933136+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:08.933136+0000","last_undegraded":"2024-09-11T05:10:08.933136+0000","last_fullsized":"2024-09-11T05:10:08.933136+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933256+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:08.933256+0000","last_peered":"2024-09-11T05:10:08.933256+0000","last_clean":"2024-09-11T05:10:08.933256+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:08.933256+0000","last_undegraded":"2024-09-11T05:10:08.933256+0000","last_fullsized":"2024-09-11T05:10:08.933256+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636040,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735800,"statfs":{"total":96636764160,"available":95985459200,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:17.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:17.937 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:17.937 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:17.950 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-11T05:10:15.954933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824236,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2822129,"kb_avail":374663124,"statfs":{"total":386547056640,"available":383655038976,"internally_reserved":0,"allocated":2048000,"data_stored":784140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2889860662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.035967"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17255,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:09.011604+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:09.011604+0000","last_peered":"2024-09-11T05:10:09.011604+0000","last_clean":"2024-09-11T05:10:09.011604+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:09.011604+0000","last_undegraded":"2024-09-11T05:10:09.011604+0000","last_fullsized":"2024-09-11T05:10:09.011604+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12915,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.934143+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:08.934143+0000","last_peered":"2024-09-11T05:10:08.934143+0000","last_clean":"2024-09-11T05:10:08.934143+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:08.934143+0000","last_undegraded":"2024-09-11T05:10:08.934143+0000","last_fullsized":"2024-09-11T05:10:08.934143+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13983,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932967+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:08.932967+0000","last_peered":"2024-09-11T05:10:08.932967+0000","last_clean":"2024-09-11T05:10:08.932967+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:08.932967+0000","last_undegraded":"2024-09-11T05:10:08.932967+0000","last_fullsized":"2024-09-11T05:10:08.932967+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17171,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929358+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:08.929358+0000","last_peered":"2024-09-11T05:10:08.929358+0000","last_clean":"2024-09-11T05:10:08.929358+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:08.929358+0000","last_undegraded":"2024-09-11T05:10:08.929358+0000","last_fullsized":"2024-09-11T05:10:08.929358+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17398,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930137+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:08.930137+0000","last_peered":"2024-09-11T05:10:08.930137+0000","last_clean":"2024-09-11T05:10:08.930137+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:08.930137+0000","last_undegraded":"2024-09-11T05:10:08.930137+0000","last_fullsized":"2024-09-11T05:10:08.930137+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17513,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933634+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:08.933634+0000","last_peered":"2024-09-11T05:10:08.933634+0000","last_clean":"2024-09-11T05:10:08.933634+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:08.933634+0000","last_undegraded":"2024-09-11T05:10:08.933634+0000","last_fullsized":"2024-09-11T05:10:08.933634+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17735,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933151+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:08.933151+0000","last_peered":"2024-09-11T05:10:08.933151+0000","last_clean":"2024-09-11T05:10:08.933151+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:08.933151+0000","last_undegraded":"2024-09-11T05:10:08.933151+0000","last_fullsized":"2024-09-11T05:10:08.933151+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17718,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930140+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:08.930140+0000","last_peered":"2024-09-11T05:10:08.930140+0000","last_clean":"2024-09-11T05:10:08.930140+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:08.930140+0000","last_undegraded":"2024-09-11T05:10:08.930140+0000","last_fullsized":"2024-09-11T05:10:08.930140+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":15584,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:15.409547+0000","last_change":"2024-09-11T05:10:07.575949+0000","last_active":"2024-09-11T05:10:15.409547+0000","last_peered":"2024-09-11T05:10:15.409547+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:15.409547+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17039,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:08.933330+0000","last_peered":"2024-09-11T05:10:08.933330+0000","last_clean":"2024-09-11T05:10:08.933330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:08.933330+0000","last_undegraded":"2024-09-11T05:10:08.933330+0000","last_fullsized":"2024-09-11T05:10:08.933330+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17553,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932973+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:08.932973+0000","last_peered":"2024-09-11T05:10:08.932973+0000","last_clean":"2024-09-11T05:10:08.932973+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:08.932973+0000","last_undegraded":"2024-09-11T05:10:08.932973+0000","last_fullsized":"2024-09-11T05:10:08.932973+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12749,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933058+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:08.933058+0000","last_peered":"2024-09-11T05:10:08.933058+0000","last_clean":"2024-09-11T05:10:08.933058+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:08.933058+0000","last_undegraded":"2024-09-11T05:10:08.933058+0000","last_fullsized":"2024-09-11T05:10:08.933058+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12562,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933136+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:08.933136+0000","last_peered":"2024-09-11T05:10:08.933136+0000","last_clean":"2024-09-11T05:10:08.933136+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:08.933136+0000","last_undegraded":"2024-09-11T05:10:08.933136+0000","last_fullsized":"2024-09-11T05:10:08.933136+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933256+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:08.933256+0000","last_peered":"2024-09-11T05:10:08.933256+0000","last_clean":"2024-09-11T05:10:08.933256+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:08.933256+0000","last_undegraded":"2024-09-11T05:10:08.933256+0000","last_fullsized":"2024-09-11T05:10:08.933256+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636040,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93735800,"statfs":{"total":96636764160,"available":95985459200,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":650768082},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:17.951 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:10:17.951 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 15584, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-11T05:10:15.409547+0000', 'last_change': '2024-09-11T05:10:07.575949+0000', 'last_active': '2024-09-11T05:10:15.409547+0000', 'last_peered': '2024-09-11T05:10:15.409547+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:10:15.409547+0000', 'last_undegraded': '2024-09-11T04:56:20.301495+0000', 'last_fullsized': '2024-09-11T04:56:20.233259+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 920, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1409, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -917}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:10:17.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:10:18.336 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:18.336 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 920/16 objects degraded (5750.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":945,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":26944,"bytes_used":2892017664,"bytes_avail":383655038976,"bytes_total":386547056640,"degraded_objects":920,"degraded_total":16,"degraded_ratio":57.5,"recovering_objects_per_sec":242,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1952,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:10:18.349 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:10:18.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:18.691 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:18.704 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-09-11T05:10:17.955225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815912,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671448,"statfs":{"total":386547056640,"available":383663562752,"internally_reserved":0,"allocated":2048000,"data_stored":784140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001590"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17255,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:09.011604+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:09.011604+0000","last_peered":"2024-09-11T05:10:09.011604+0000","last_clean":"2024-09-11T05:10:09.011604+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:09.011604+0000","last_undegraded":"2024-09-11T05:10:09.011604+0000","last_fullsized":"2024-09-11T05:10:09.011604+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12915,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.934143+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:08.934143+0000","last_peered":"2024-09-11T05:10:08.934143+0000","last_clean":"2024-09-11T05:10:08.934143+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:08.934143+0000","last_undegraded":"2024-09-11T05:10:08.934143+0000","last_fullsized":"2024-09-11T05:10:08.934143+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13983,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932967+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:08.932967+0000","last_peered":"2024-09-11T05:10:08.932967+0000","last_clean":"2024-09-11T05:10:08.932967+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:08.932967+0000","last_undegraded":"2024-09-11T05:10:08.932967+0000","last_fullsized":"2024-09-11T05:10:08.932967+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17171,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929358+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:08.929358+0000","last_peered":"2024-09-11T05:10:08.929358+0000","last_clean":"2024-09-11T05:10:08.929358+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:08.929358+0000","last_undegraded":"2024-09-11T05:10:08.929358+0000","last_fullsized":"2024-09-11T05:10:08.929358+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17398,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930137+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:08.930137+0000","last_peered":"2024-09-11T05:10:08.930137+0000","last_clean":"2024-09-11T05:10:08.930137+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:08.930137+0000","last_undegraded":"2024-09-11T05:10:08.930137+0000","last_fullsized":"2024-09-11T05:10:08.930137+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17513,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933634+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:08.933634+0000","last_peered":"2024-09-11T05:10:08.933634+0000","last_clean":"2024-09-11T05:10:08.933634+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:08.933634+0000","last_undegraded":"2024-09-11T05:10:08.933634+0000","last_fullsized":"2024-09-11T05:10:08.933634+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17735,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933151+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:08.933151+0000","last_peered":"2024-09-11T05:10:08.933151+0000","last_clean":"2024-09-11T05:10:08.933151+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:08.933151+0000","last_undegraded":"2024-09-11T05:10:08.933151+0000","last_fullsized":"2024-09-11T05:10:08.933151+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17718,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.930140+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:08.930140+0000","last_peered":"2024-09-11T05:10:08.930140+0000","last_clean":"2024-09-11T05:10:08.930140+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:08.930140+0000","last_undegraded":"2024-09-11T05:10:08.930140+0000","last_fullsized":"2024-09-11T05:10:08.930140+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":15584,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-11T05:10:15.409547+0000","last_change":"2024-09-11T05:10:07.575949+0000","last_active":"2024-09-11T05:10:15.409547+0000","last_peered":"2024-09-11T05:10:15.409547+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:15.409547+0000","last_undegraded":"2024-09-11T04:56:20.301495+0000","last_fullsized":"2024-09-11T04:56:20.233259+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17039,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:08.933330+0000","last_peered":"2024-09-11T05:10:08.933330+0000","last_clean":"2024-09-11T05:10:08.933330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:08.933330+0000","last_undegraded":"2024-09-11T05:10:08.933330+0000","last_fullsized":"2024-09-11T05:10:08.933330+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17553,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932973+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:08.932973+0000","last_peered":"2024-09-11T05:10:08.932973+0000","last_clean":"2024-09-11T05:10:08.932973+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:08.932973+0000","last_undegraded":"2024-09-11T05:10:08.932973+0000","last_fullsized":"2024-09-11T05:10:08.932973+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12749,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933058+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:08.933058+0000","last_peered":"2024-09-11T05:10:08.933058+0000","last_clean":"2024-09-11T05:10:08.933058+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:08.933058+0000","last_undegraded":"2024-09-11T05:10:08.933058+0000","last_fullsized":"2024-09-11T05:10:08.933058+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12562,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933136+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:08.933136+0000","last_peered":"2024-09-11T05:10:08.933136+0000","last_clean":"2024-09-11T05:10:08.933136+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:08.933136+0000","last_undegraded":"2024-09-11T05:10:08.933136+0000","last_fullsized":"2024-09-11T05:10:08.933136+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933256+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:08.933256+0000","last_peered":"2024-09-11T05:10:08.933256+0000","last_clean":"2024-09-11T05:10:08.933256+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:08.933256+0000","last_undegraded":"2024-09-11T05:10:08.933256+0000","last_fullsized":"2024-09-11T05:10:08.933256+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627716,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744124,"statfs":{"total":96636764160,"available":95993982976,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:21.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:22.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:22.022 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-09-11T05:10:21.955900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815912,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671448,"statfs":{"total":386547056640,"available":383663562752,"internally_reserved":0,"allocated":2048000,"data_stored":784140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001303"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17257,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.375261+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:20.375261+0000","last_peered":"2024-09-11T05:10:20.375261+0000","last_clean":"2024-09-11T05:10:20.375261+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:20.375261+0000","last_undegraded":"2024-09-11T05:10:20.375261+0000","last_fullsized":"2024-09-11T05:10:20.375261+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12915,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.934143+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:08.934143+0000","last_peered":"2024-09-11T05:10:08.934143+0000","last_clean":"2024-09-11T05:10:08.934143+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:08.934143+0000","last_undegraded":"2024-09-11T05:10:08.934143+0000","last_fullsized":"2024-09-11T05:10:08.934143+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13983,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932967+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:08.932967+0000","last_peered":"2024-09-11T05:10:08.932967+0000","last_clean":"2024-09-11T05:10:08.932967+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:08.932967+0000","last_undegraded":"2024-09-11T05:10:08.932967+0000","last_fullsized":"2024-09-11T05:10:08.932967+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17173,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.375301+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:20.375301+0000","last_peered":"2024-09-11T05:10:20.375301+0000","last_clean":"2024-09-11T05:10:20.375301+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:20.375301+0000","last_undegraded":"2024-09-11T05:10:20.375301+0000","last_fullsized":"2024-09-11T05:10:20.375301+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17400,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.376270+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:20.376270+0000","last_peered":"2024-09-11T05:10:20.376270+0000","last_clean":"2024-09-11T05:10:20.376270+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:20.376270+0000","last_undegraded":"2024-09-11T05:10:20.376270+0000","last_fullsized":"2024-09-11T05:10:20.376270+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17513,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933634+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:08.933634+0000","last_peered":"2024-09-11T05:10:08.933634+0000","last_clean":"2024-09-11T05:10:08.933634+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:08.933634+0000","last_undegraded":"2024-09-11T05:10:08.933634+0000","last_fullsized":"2024-09-11T05:10:08.933634+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17735,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933151+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:08.933151+0000","last_peered":"2024-09-11T05:10:08.933151+0000","last_clean":"2024-09-11T05:10:08.933151+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:08.933151+0000","last_undegraded":"2024-09-11T05:10:08.933151+0000","last_fullsized":"2024-09-11T05:10:08.933151+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17720,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.376134+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:20.376134+0000","last_peered":"2024-09-11T05:10:20.376134+0000","last_clean":"2024-09-11T05:10:20.376134+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:20.376134+0000","last_undegraded":"2024-09-11T05:10:20.376134+0000","last_fullsized":"2024-09-11T05:10:20.376134+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17427,"reported_epoch":62,"state":"peering","last_fresh":"2024-09-11T05:10:20.380260+0000","last_change":"2024-09-11T05:10:20.380260+0000","last_active":"2024-09-11T05:10:20.380260+0000","last_peered":"2024-09-11T05:10:19.457274+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:20.380260+0000","last_undegraded":"2024-09-11T05:10:20.380260+0000","last_fullsized":"2024-09-11T05:10:20.380260+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2329,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17039,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:08.933330+0000","last_peered":"2024-09-11T05:10:08.933330+0000","last_clean":"2024-09-11T05:10:08.933330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:08.933330+0000","last_undegraded":"2024-09-11T05:10:08.933330+0000","last_fullsized":"2024-09-11T05:10:08.933330+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17553,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932973+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:08.932973+0000","last_peered":"2024-09-11T05:10:08.932973+0000","last_clean":"2024-09-11T05:10:08.932973+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:08.932973+0000","last_undegraded":"2024-09-11T05:10:08.932973+0000","last_fullsized":"2024-09-11T05:10:08.932973+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12749,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933058+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:08.933058+0000","last_peered":"2024-09-11T05:10:08.933058+0000","last_clean":"2024-09-11T05:10:08.933058+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:08.933058+0000","last_undegraded":"2024-09-11T05:10:08.933058+0000","last_fullsized":"2024-09-11T05:10:08.933058+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12562,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933136+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:08.933136+0000","last_peered":"2024-09-11T05:10:08.933136+0000","last_clean":"2024-09-11T05:10:08.933136+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:08.933136+0000","last_undegraded":"2024-09-11T05:10:08.933136+0000","last_fullsized":"2024-09-11T05:10:08.933136+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933256+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:08.933256+0000","last_peered":"2024-09-11T05:10:08.933256+0000","last_clean":"2024-09-11T05:10:08.933256+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:08.933256+0000","last_undegraded":"2024-09-11T05:10:08.933256+0000","last_fullsized":"2024-09-11T05:10:08.933256+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627716,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744124,"statfs":{"total":96636764160,"available":95993982976,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:22.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:22.319 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:22.320 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:22.331 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-09-11T05:10:21.955900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815912,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671448,"statfs":{"total":386547056640,"available":383663562752,"internally_reserved":0,"allocated":2048000,"data_stored":784140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001303"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17257,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.375261+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:20.375261+0000","last_peered":"2024-09-11T05:10:20.375261+0000","last_clean":"2024-09-11T05:10:20.375261+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:20.375261+0000","last_undegraded":"2024-09-11T05:10:20.375261+0000","last_fullsized":"2024-09-11T05:10:20.375261+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12915,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.934143+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:08.934143+0000","last_peered":"2024-09-11T05:10:08.934143+0000","last_clean":"2024-09-11T05:10:08.934143+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:08.934143+0000","last_undegraded":"2024-09-11T05:10:08.934143+0000","last_fullsized":"2024-09-11T05:10:08.934143+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13983,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932967+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:08.932967+0000","last_peered":"2024-09-11T05:10:08.932967+0000","last_clean":"2024-09-11T05:10:08.932967+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:08.932967+0000","last_undegraded":"2024-09-11T05:10:08.932967+0000","last_fullsized":"2024-09-11T05:10:08.932967+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17173,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.375301+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:20.375301+0000","last_peered":"2024-09-11T05:10:20.375301+0000","last_clean":"2024-09-11T05:10:20.375301+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:20.375301+0000","last_undegraded":"2024-09-11T05:10:20.375301+0000","last_fullsized":"2024-09-11T05:10:20.375301+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17400,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.376270+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:20.376270+0000","last_peered":"2024-09-11T05:10:20.376270+0000","last_clean":"2024-09-11T05:10:20.376270+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:20.376270+0000","last_undegraded":"2024-09-11T05:10:20.376270+0000","last_fullsized":"2024-09-11T05:10:20.376270+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17513,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933634+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:08.933634+0000","last_peered":"2024-09-11T05:10:08.933634+0000","last_clean":"2024-09-11T05:10:08.933634+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:08.933634+0000","last_undegraded":"2024-09-11T05:10:08.933634+0000","last_fullsized":"2024-09-11T05:10:08.933634+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17735,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933151+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:08.933151+0000","last_peered":"2024-09-11T05:10:08.933151+0000","last_clean":"2024-09-11T05:10:08.933151+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:08.933151+0000","last_undegraded":"2024-09-11T05:10:08.933151+0000","last_fullsized":"2024-09-11T05:10:08.933151+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17720,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.376134+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:20.376134+0000","last_peered":"2024-09-11T05:10:20.376134+0000","last_clean":"2024-09-11T05:10:20.376134+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:20.376134+0000","last_undegraded":"2024-09-11T05:10:20.376134+0000","last_fullsized":"2024-09-11T05:10:20.376134+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17427,"reported_epoch":62,"state":"peering","last_fresh":"2024-09-11T05:10:20.380260+0000","last_change":"2024-09-11T05:10:20.380260+0000","last_active":"2024-09-11T05:10:20.380260+0000","last_peered":"2024-09-11T05:10:19.457274+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:20.380260+0000","last_undegraded":"2024-09-11T05:10:20.380260+0000","last_fullsized":"2024-09-11T05:10:20.380260+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2329,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17039,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:08.933330+0000","last_peered":"2024-09-11T05:10:08.933330+0000","last_clean":"2024-09-11T05:10:08.933330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:08.933330+0000","last_undegraded":"2024-09-11T05:10:08.933330+0000","last_fullsized":"2024-09-11T05:10:08.933330+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17553,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932973+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:08.932973+0000","last_peered":"2024-09-11T05:10:08.932973+0000","last_clean":"2024-09-11T05:10:08.932973+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:08.932973+0000","last_undegraded":"2024-09-11T05:10:08.932973+0000","last_fullsized":"2024-09-11T05:10:08.932973+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12749,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933058+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:08.933058+0000","last_peered":"2024-09-11T05:10:08.933058+0000","last_clean":"2024-09-11T05:10:08.933058+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:08.933058+0000","last_undegraded":"2024-09-11T05:10:08.933058+0000","last_fullsized":"2024-09-11T05:10:08.933058+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12562,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933136+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:08.933136+0000","last_peered":"2024-09-11T05:10:08.933136+0000","last_clean":"2024-09-11T05:10:08.933136+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:08.933136+0000","last_undegraded":"2024-09-11T05:10:08.933136+0000","last_fullsized":"2024-09-11T05:10:08.933136+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933256+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:08.933256+0000","last_peered":"2024-09-11T05:10:08.933256+0000","last_clean":"2024-09-11T05:10:08.933256+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:08.933256+0000","last_undegraded":"2024-09-11T05:10:08.933256+0000","last_fullsized":"2024-09-11T05:10:08.933256+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627716,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744124,"statfs":{"total":96636764160,"available":95993982976,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:22.332 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-11T05:10:22.333 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12650", 'reported_seq': 17427, 'reported_epoch': 62, 'state': 'peering', 'last_fresh': '2024-09-11T05:10:20.380260+0000', 'last_change': '2024-09-11T05:10:20.380260+0000', 'last_active': '2024-09-11T05:10:20.380260+0000', 'last_peered': '2024-09-11T05:10:19.457274+0000', 'last_clean': '2024-09-11T04:55:08.615108+0000', 'last_became_active': '2024-09-11T04:56:20.648867+0000', 'last_became_peered': '2024-09-11T04:56:20.648867+0000', 'last_unstale': '2024-09-11T05:10:20.380260+0000', 'last_undegraded': '2024-09-11T05:10:20.380260+0000', 'last_fullsized': '2024-09-11T05:10:20.380260+0000', 'mapping_epoch': 62, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'last_clean_scrub_stamp': '2024-09-11T04:54:56.683804+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20208, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 50596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2329, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T05:10:22.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T05:10:22.711 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:22.711 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":949,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1726030578,"num_in_osds":4,"osd_in_since":1726030474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":26944,"bytes_used":2883493888,"bytes_avail":383663562752,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"recovering_objects_per_sec":223,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1785,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-11T04:54:33:002250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-11T05:08:03.932168+0000","services":{}},"progress_events":{}} 2024-09-11T05:10:22.724 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-11T05:10:22.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:23.020 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:23.032 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-09-11T05:10:21.955900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815912,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2813809,"kb_avail":374671448,"statfs":{"total":386547056640,"available":383663562752,"internally_reserved":0,"allocated":2048000,"data_stored":784140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2881340982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001303"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17257,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.375261+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:20.375261+0000","last_peered":"2024-09-11T05:10:20.375261+0000","last_clean":"2024-09-11T05:10:20.375261+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:20.375261+0000","last_undegraded":"2024-09-11T05:10:20.375261+0000","last_fullsized":"2024-09-11T05:10:20.375261+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12915,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.934143+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:08.934143+0000","last_peered":"2024-09-11T05:10:08.934143+0000","last_clean":"2024-09-11T05:10:08.934143+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:08.934143+0000","last_undegraded":"2024-09-11T05:10:08.934143+0000","last_fullsized":"2024-09-11T05:10:08.934143+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16761,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929887+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:08.929887+0000","last_peered":"2024-09-11T05:10:08.929887+0000","last_clean":"2024-09-11T05:10:08.929887+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:08.929887+0000","last_undegraded":"2024-09-11T05:10:08.929887+0000","last_fullsized":"2024-09-11T05:10:08.929887+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13983,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932967+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:08.932967+0000","last_peered":"2024-09-11T05:10:08.932967+0000","last_clean":"2024-09-11T05:10:08.932967+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:08.932967+0000","last_undegraded":"2024-09-11T05:10:08.932967+0000","last_fullsized":"2024-09-11T05:10:08.932967+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17173,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.375301+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:20.375301+0000","last_peered":"2024-09-11T05:10:20.375301+0000","last_clean":"2024-09-11T05:10:20.375301+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:20.375301+0000","last_undegraded":"2024-09-11T05:10:20.375301+0000","last_fullsized":"2024-09-11T05:10:20.375301+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17400,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.376270+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:20.376270+0000","last_peered":"2024-09-11T05:10:20.376270+0000","last_clean":"2024-09-11T05:10:20.376270+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:20.376270+0000","last_undegraded":"2024-09-11T05:10:20.376270+0000","last_fullsized":"2024-09-11T05:10:20.376270+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17513,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933634+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:08.933634+0000","last_peered":"2024-09-11T05:10:08.933634+0000","last_clean":"2024-09-11T05:10:08.933634+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:08.933634+0000","last_undegraded":"2024-09-11T05:10:08.933634+0000","last_fullsized":"2024-09-11T05:10:08.933634+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17735,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933151+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:08.933151+0000","last_peered":"2024-09-11T05:10:08.933151+0000","last_clean":"2024-09-11T05:10:08.933151+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:08.933151+0000","last_undegraded":"2024-09-11T05:10:08.933151+0000","last_fullsized":"2024-09-11T05:10:08.933151+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17720,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T05:10:20.376134+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:20.376134+0000","last_peered":"2024-09-11T05:10:20.376134+0000","last_clean":"2024-09-11T05:10:20.376134+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:20.376134+0000","last_undegraded":"2024-09-11T05:10:20.376134+0000","last_fullsized":"2024-09-11T05:10:20.376134+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17327,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.929774+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:08.929774+0000","last_peered":"2024-09-11T05:10:08.929774+0000","last_clean":"2024-09-11T05:10:08.929774+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:08.929774+0000","last_undegraded":"2024-09-11T05:10:08.929774+0000","last_fullsized":"2024-09-11T05:10:08.929774+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17427,"reported_epoch":62,"state":"peering","last_fresh":"2024-09-11T05:10:20.380260+0000","last_change":"2024-09-11T05:10:20.380260+0000","last_active":"2024-09-11T05:10:20.380260+0000","last_peered":"2024-09-11T05:10:19.457274+0000","last_clean":"2024-09-11T04:55:08.615108+0000","last_became_active":"2024-09-11T04:56:20.648867+0000","last_became_peered":"2024-09-11T04:56:20.648867+0000","last_unstale":"2024-09-11T05:10:20.380260+0000","last_undegraded":"2024-09-11T05:10:20.380260+0000","last_fullsized":"2024-09-11T05:10:20.380260+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20208,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2329,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17039,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933330+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:08.933330+0000","last_peered":"2024-09-11T05:10:08.933330+0000","last_clean":"2024-09-11T05:10:08.933330+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:08.933330+0000","last_undegraded":"2024-09-11T05:10:08.933330+0000","last_fullsized":"2024-09-11T05:10:08.933330+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17553,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.932973+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:08.932973+0000","last_peered":"2024-09-11T05:10:08.932973+0000","last_clean":"2024-09-11T05:10:08.932973+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:08.932973+0000","last_undegraded":"2024-09-11T05:10:08.932973+0000","last_fullsized":"2024-09-11T05:10:08.932973+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12749,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933058+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:08.933058+0000","last_peered":"2024-09-11T05:10:08.933058+0000","last_clean":"2024-09-11T05:10:08.933058+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:08.933058+0000","last_undegraded":"2024-09-11T05:10:08.933058+0000","last_fullsized":"2024-09-11T05:10:08.933058+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12562,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933136+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:08.933136+0000","last_peered":"2024-09-11T05:10:08.933136+0000","last_clean":"2024-09-11T05:10:08.933136+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:08.933136+0000","last_undegraded":"2024-09-11T05:10:08.933136+0000","last_fullsized":"2024-09-11T05:10:08.933136+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T05:10:08.933256+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:08.933256+0000","last_peered":"2024-09-11T05:10:08.933256+0000","last_clean":"2024-09-11T05:10:08.933256+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:08.933256+0000","last_undegraded":"2024-09-11T05:10:08.933256+0000","last_fullsized":"2024-09-11T05:10:08.933256+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26944,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48397,"num_write_kb":151236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744396,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627444,"statfs":{"total":96636764160,"available":95874502656,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627716,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93744124,"statfs":{"total":96636764160,"available":95993982976,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":642248402},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:26.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:26.345 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:26.357 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-09-11T05:10:25.956535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838172,"kb_used_data":2056,"kb_used_omap":14,"kb_used_meta":2836017,"kb_avail":374649188,"statfs":{"total":386547056640,"available":383640768512,"internally_reserved":0,"allocated":2105344,"data_stored":805553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2904081974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12647,"num_write_kb":-50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001309"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17259,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379419+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:21.379419+0000","last_peered":"2024-09-11T05:10:21.379419+0000","last_clean":"2024-09-11T05:10:21.379419+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:21.379419+0000","last_undegraded":"2024-09-11T05:10:21.379419+0000","last_fullsized":"2024-09-11T05:10:21.379419+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12919,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376575+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:21.376575+0000","last_peered":"2024-09-11T05:10:21.376575+0000","last_clean":"2024-09-11T05:10:21.376575+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:21.376575+0000","last_undegraded":"2024-09-11T05:10:21.376575+0000","last_fullsized":"2024-09-11T05:10:21.376575+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16765,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376550+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:21.376550+0000","last_peered":"2024-09-11T05:10:21.376550+0000","last_clean":"2024-09-11T05:10:21.376550+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:21.376550+0000","last_undegraded":"2024-09-11T05:10:21.376550+0000","last_fullsized":"2024-09-11T05:10:21.376550+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13987,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.375920+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:21.375920+0000","last_peered":"2024-09-11T05:10:21.375920+0000","last_clean":"2024-09-11T05:10:21.375920+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:21.375920+0000","last_undegraded":"2024-09-11T05:10:21.375920+0000","last_fullsized":"2024-09-11T05:10:21.375920+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17175,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379145+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:21.379145+0000","last_peered":"2024-09-11T05:10:21.379145+0000","last_clean":"2024-09-11T05:10:21.379145+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:21.379145+0000","last_undegraded":"2024-09-11T05:10:21.379145+0000","last_fullsized":"2024-09-11T05:10:21.379145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17402,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379352+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:21.379352+0000","last_peered":"2024-09-11T05:10:21.379352+0000","last_clean":"2024-09-11T05:10:21.379352+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:21.379352+0000","last_undegraded":"2024-09-11T05:10:21.379352+0000","last_fullsized":"2024-09-11T05:10:21.379352+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17517,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.382010+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:21.382010+0000","last_peered":"2024-09-11T05:10:21.382010+0000","last_clean":"2024-09-11T05:10:21.382010+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:21.382010+0000","last_undegraded":"2024-09-11T05:10:21.382010+0000","last_fullsized":"2024-09-11T05:10:21.382010+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17739,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376986+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:21.376986+0000","last_peered":"2024-09-11T05:10:21.376986+0000","last_clean":"2024-09-11T05:10:21.376986+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:21.376986+0000","last_undegraded":"2024-09-11T05:10:21.376986+0000","last_fullsized":"2024-09-11T05:10:21.376986+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17722,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379203+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:21.379203+0000","last_peered":"2024-09-11T05:10:21.379203+0000","last_clean":"2024-09-11T05:10:21.379203+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:21.379203+0000","last_undegraded":"2024-09-11T05:10:21.379203+0000","last_fullsized":"2024-09-11T05:10:21.379203+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17331,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376712+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:21.376712+0000","last_peered":"2024-09-11T05:10:21.376712+0000","last_clean":"2024-09-11T05:10:21.376712+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:21.376712+0000","last_undegraded":"2024-09-11T05:10:21.376712+0000","last_fullsized":"2024-09-11T05:10:21.376712+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17446,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:22.452891+0000","last_change":"2024-09-11T05:10:22.452891+0000","last_active":"2024-09-11T05:10:22.452891+0000","last_peered":"2024-09-11T05:10:22.452891+0000","last_clean":"2024-09-11T05:10:22.452891+0000","last_became_active":"2024-09-11T05:10:21.382743+0000","last_became_peered":"2024-09-11T05:10:21.382743+0000","last_unstale":"2024-09-11T05:10:22.452891+0000","last_undegraded":"2024-09-11T05:10:22.452891+0000","last_fullsized":"2024-09-11T05:10:22.452891+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12650","last_scrub_stamp":"2024-09-11T05:10:22.452806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:10:22.452806+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:52.349912+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17043,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377533+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:21.377533+0000","last_peered":"2024-09-11T05:10:21.377533+0000","last_clean":"2024-09-11T05:10:21.377533+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:21.377533+0000","last_undegraded":"2024-09-11T05:10:21.377533+0000","last_fullsized":"2024-09-11T05:10:21.377533+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376078+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:21.376078+0000","last_peered":"2024-09-11T05:10:21.376078+0000","last_clean":"2024-09-11T05:10:21.376078+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:21.376078+0000","last_undegraded":"2024-09-11T05:10:21.376078+0000","last_fullsized":"2024-09-11T05:10:21.376078+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12753,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376560+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:21.376560+0000","last_peered":"2024-09-11T05:10:21.376560+0000","last_clean":"2024-09-11T05:10:21.376560+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:21.376560+0000","last_undegraded":"2024-09-11T05:10:21.376560+0000","last_fullsized":"2024-09-11T05:10:21.376560+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12566,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376819+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:21.376819+0000","last_peered":"2024-09-11T05:10:21.376819+0000","last_clean":"2024-09-11T05:10:21.376819+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:21.376819+0000","last_undegraded":"2024-09-11T05:10:21.376819+0000","last_fullsized":"2024-09-11T05:10:21.376819+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12947,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377121+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:21.377121+0000","last_peered":"2024-09-11T05:10:21.377121+0000","last_clean":"2024-09-11T05:10:21.377121+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:21.377121+0000","last_undegraded":"2024-09-11T05:10:21.377121+0000","last_fullsized":"2024-09-11T05:10:21.377121+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744404,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627436,"statfs":{"total":96636764160,"available":95874494464,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721692,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650148,"statfs":{"total":96636764160,"available":95897751552,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649936,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93721904,"statfs":{"total":96636764160,"available":95971229696,"internally_reserved":0,"allocated":520192,"data_stored":199103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":664989394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:26.358 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-11T05:10:26.358 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-11T05:10:26.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:26.657 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:26.658 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:26.669 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-09-11T05:10:25.956535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838172,"kb_used_data":2056,"kb_used_omap":14,"kb_used_meta":2836017,"kb_avail":374649188,"statfs":{"total":386547056640,"available":383640768512,"internally_reserved":0,"allocated":2105344,"data_stored":805553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2904081974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12647,"num_write_kb":-50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001309"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17259,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379419+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:21.379419+0000","last_peered":"2024-09-11T05:10:21.379419+0000","last_clean":"2024-09-11T05:10:21.379419+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:21.379419+0000","last_undegraded":"2024-09-11T05:10:21.379419+0000","last_fullsized":"2024-09-11T05:10:21.379419+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12919,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376575+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:21.376575+0000","last_peered":"2024-09-11T05:10:21.376575+0000","last_clean":"2024-09-11T05:10:21.376575+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:21.376575+0000","last_undegraded":"2024-09-11T05:10:21.376575+0000","last_fullsized":"2024-09-11T05:10:21.376575+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16765,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376550+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:21.376550+0000","last_peered":"2024-09-11T05:10:21.376550+0000","last_clean":"2024-09-11T05:10:21.376550+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:21.376550+0000","last_undegraded":"2024-09-11T05:10:21.376550+0000","last_fullsized":"2024-09-11T05:10:21.376550+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13987,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.375920+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:21.375920+0000","last_peered":"2024-09-11T05:10:21.375920+0000","last_clean":"2024-09-11T05:10:21.375920+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:21.375920+0000","last_undegraded":"2024-09-11T05:10:21.375920+0000","last_fullsized":"2024-09-11T05:10:21.375920+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17175,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379145+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:21.379145+0000","last_peered":"2024-09-11T05:10:21.379145+0000","last_clean":"2024-09-11T05:10:21.379145+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:21.379145+0000","last_undegraded":"2024-09-11T05:10:21.379145+0000","last_fullsized":"2024-09-11T05:10:21.379145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17402,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379352+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:21.379352+0000","last_peered":"2024-09-11T05:10:21.379352+0000","last_clean":"2024-09-11T05:10:21.379352+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:21.379352+0000","last_undegraded":"2024-09-11T05:10:21.379352+0000","last_fullsized":"2024-09-11T05:10:21.379352+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17517,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.382010+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:21.382010+0000","last_peered":"2024-09-11T05:10:21.382010+0000","last_clean":"2024-09-11T05:10:21.382010+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:21.382010+0000","last_undegraded":"2024-09-11T05:10:21.382010+0000","last_fullsized":"2024-09-11T05:10:21.382010+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17739,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376986+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:21.376986+0000","last_peered":"2024-09-11T05:10:21.376986+0000","last_clean":"2024-09-11T05:10:21.376986+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:21.376986+0000","last_undegraded":"2024-09-11T05:10:21.376986+0000","last_fullsized":"2024-09-11T05:10:21.376986+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17722,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379203+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:21.379203+0000","last_peered":"2024-09-11T05:10:21.379203+0000","last_clean":"2024-09-11T05:10:21.379203+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:21.379203+0000","last_undegraded":"2024-09-11T05:10:21.379203+0000","last_fullsized":"2024-09-11T05:10:21.379203+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17331,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376712+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:21.376712+0000","last_peered":"2024-09-11T05:10:21.376712+0000","last_clean":"2024-09-11T05:10:21.376712+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:21.376712+0000","last_undegraded":"2024-09-11T05:10:21.376712+0000","last_fullsized":"2024-09-11T05:10:21.376712+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17446,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:22.452891+0000","last_change":"2024-09-11T05:10:22.452891+0000","last_active":"2024-09-11T05:10:22.452891+0000","last_peered":"2024-09-11T05:10:22.452891+0000","last_clean":"2024-09-11T05:10:22.452891+0000","last_became_active":"2024-09-11T05:10:21.382743+0000","last_became_peered":"2024-09-11T05:10:21.382743+0000","last_unstale":"2024-09-11T05:10:22.452891+0000","last_undegraded":"2024-09-11T05:10:22.452891+0000","last_fullsized":"2024-09-11T05:10:22.452891+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12650","last_scrub_stamp":"2024-09-11T05:10:22.452806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:10:22.452806+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:52.349912+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17043,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377533+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:21.377533+0000","last_peered":"2024-09-11T05:10:21.377533+0000","last_clean":"2024-09-11T05:10:21.377533+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:21.377533+0000","last_undegraded":"2024-09-11T05:10:21.377533+0000","last_fullsized":"2024-09-11T05:10:21.377533+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376078+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:21.376078+0000","last_peered":"2024-09-11T05:10:21.376078+0000","last_clean":"2024-09-11T05:10:21.376078+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:21.376078+0000","last_undegraded":"2024-09-11T05:10:21.376078+0000","last_fullsized":"2024-09-11T05:10:21.376078+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12753,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376560+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:21.376560+0000","last_peered":"2024-09-11T05:10:21.376560+0000","last_clean":"2024-09-11T05:10:21.376560+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:21.376560+0000","last_undegraded":"2024-09-11T05:10:21.376560+0000","last_fullsized":"2024-09-11T05:10:21.376560+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12566,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376819+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:21.376819+0000","last_peered":"2024-09-11T05:10:21.376819+0000","last_clean":"2024-09-11T05:10:21.376819+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:21.376819+0000","last_undegraded":"2024-09-11T05:10:21.376819+0000","last_fullsized":"2024-09-11T05:10:21.376819+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12947,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377121+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:21.377121+0000","last_peered":"2024-09-11T05:10:21.377121+0000","last_clean":"2024-09-11T05:10:21.377121+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:21.377121+0000","last_undegraded":"2024-09-11T05:10:21.377121+0000","last_fullsized":"2024-09-11T05:10:21.377121+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744404,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627436,"statfs":{"total":96636764160,"available":95874494464,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721692,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650148,"statfs":{"total":96636764160,"available":95897751552,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649936,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93721904,"statfs":{"total":96636764160,"available":95971229696,"internally_reserved":0,"allocated":520192,"data_stored":199103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":664989394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:26.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:26.968 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:26.980 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-09-11T05:10:25.956535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838172,"kb_used_data":2056,"kb_used_omap":14,"kb_used_meta":2836017,"kb_avail":374649188,"statfs":{"total":386547056640,"available":383640768512,"internally_reserved":0,"allocated":2105344,"data_stored":805553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2904081974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12647,"num_write_kb":-50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001309"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17259,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379419+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:21.379419+0000","last_peered":"2024-09-11T05:10:21.379419+0000","last_clean":"2024-09-11T05:10:21.379419+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:21.379419+0000","last_undegraded":"2024-09-11T05:10:21.379419+0000","last_fullsized":"2024-09-11T05:10:21.379419+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12919,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376575+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:21.376575+0000","last_peered":"2024-09-11T05:10:21.376575+0000","last_clean":"2024-09-11T05:10:21.376575+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:21.376575+0000","last_undegraded":"2024-09-11T05:10:21.376575+0000","last_fullsized":"2024-09-11T05:10:21.376575+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16765,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376550+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:21.376550+0000","last_peered":"2024-09-11T05:10:21.376550+0000","last_clean":"2024-09-11T05:10:21.376550+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:21.376550+0000","last_undegraded":"2024-09-11T05:10:21.376550+0000","last_fullsized":"2024-09-11T05:10:21.376550+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13987,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.375920+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:21.375920+0000","last_peered":"2024-09-11T05:10:21.375920+0000","last_clean":"2024-09-11T05:10:21.375920+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:21.375920+0000","last_undegraded":"2024-09-11T05:10:21.375920+0000","last_fullsized":"2024-09-11T05:10:21.375920+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17175,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379145+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:21.379145+0000","last_peered":"2024-09-11T05:10:21.379145+0000","last_clean":"2024-09-11T05:10:21.379145+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:21.379145+0000","last_undegraded":"2024-09-11T05:10:21.379145+0000","last_fullsized":"2024-09-11T05:10:21.379145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17402,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379352+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:21.379352+0000","last_peered":"2024-09-11T05:10:21.379352+0000","last_clean":"2024-09-11T05:10:21.379352+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:21.379352+0000","last_undegraded":"2024-09-11T05:10:21.379352+0000","last_fullsized":"2024-09-11T05:10:21.379352+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17517,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.382010+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:21.382010+0000","last_peered":"2024-09-11T05:10:21.382010+0000","last_clean":"2024-09-11T05:10:21.382010+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:21.382010+0000","last_undegraded":"2024-09-11T05:10:21.382010+0000","last_fullsized":"2024-09-11T05:10:21.382010+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17739,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376986+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:21.376986+0000","last_peered":"2024-09-11T05:10:21.376986+0000","last_clean":"2024-09-11T05:10:21.376986+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:21.376986+0000","last_undegraded":"2024-09-11T05:10:21.376986+0000","last_fullsized":"2024-09-11T05:10:21.376986+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17722,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379203+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:21.379203+0000","last_peered":"2024-09-11T05:10:21.379203+0000","last_clean":"2024-09-11T05:10:21.379203+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:21.379203+0000","last_undegraded":"2024-09-11T05:10:21.379203+0000","last_fullsized":"2024-09-11T05:10:21.379203+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17331,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376712+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:21.376712+0000","last_peered":"2024-09-11T05:10:21.376712+0000","last_clean":"2024-09-11T05:10:21.376712+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:21.376712+0000","last_undegraded":"2024-09-11T05:10:21.376712+0000","last_fullsized":"2024-09-11T05:10:21.376712+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17446,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:22.452891+0000","last_change":"2024-09-11T05:10:22.452891+0000","last_active":"2024-09-11T05:10:22.452891+0000","last_peered":"2024-09-11T05:10:22.452891+0000","last_clean":"2024-09-11T05:10:22.452891+0000","last_became_active":"2024-09-11T05:10:21.382743+0000","last_became_peered":"2024-09-11T05:10:21.382743+0000","last_unstale":"2024-09-11T05:10:22.452891+0000","last_undegraded":"2024-09-11T05:10:22.452891+0000","last_fullsized":"2024-09-11T05:10:22.452891+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12650","last_scrub_stamp":"2024-09-11T05:10:22.452806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:10:22.452806+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:52.349912+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17043,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377533+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:21.377533+0000","last_peered":"2024-09-11T05:10:21.377533+0000","last_clean":"2024-09-11T05:10:21.377533+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:21.377533+0000","last_undegraded":"2024-09-11T05:10:21.377533+0000","last_fullsized":"2024-09-11T05:10:21.377533+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376078+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:21.376078+0000","last_peered":"2024-09-11T05:10:21.376078+0000","last_clean":"2024-09-11T05:10:21.376078+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:21.376078+0000","last_undegraded":"2024-09-11T05:10:21.376078+0000","last_fullsized":"2024-09-11T05:10:21.376078+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12753,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376560+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:21.376560+0000","last_peered":"2024-09-11T05:10:21.376560+0000","last_clean":"2024-09-11T05:10:21.376560+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:21.376560+0000","last_undegraded":"2024-09-11T05:10:21.376560+0000","last_fullsized":"2024-09-11T05:10:21.376560+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12566,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376819+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:21.376819+0000","last_peered":"2024-09-11T05:10:21.376819+0000","last_clean":"2024-09-11T05:10:21.376819+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:21.376819+0000","last_undegraded":"2024-09-11T05:10:21.376819+0000","last_fullsized":"2024-09-11T05:10:21.376819+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12947,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377121+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:21.377121+0000","last_peered":"2024-09-11T05:10:21.377121+0000","last_clean":"2024-09-11T05:10:21.377121+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:21.377121+0000","last_undegraded":"2024-09-11T05:10:21.377121+0000","last_fullsized":"2024-09-11T05:10:21.377121+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744404,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627436,"statfs":{"total":96636764160,"available":95874494464,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721692,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650148,"statfs":{"total":96636764160,"available":95897751552,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649936,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93721904,"statfs":{"total":96636764160,"available":95971229696,"internally_reserved":0,"allocated":520192,"data_stored":199103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":664989394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:26.981 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-11T05:10:26.981 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T05:10:26.990 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T05:10:26.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:27.284 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:27.296 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-09-11T05:10:25.956535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838172,"kb_used_data":2056,"kb_used_omap":14,"kb_used_meta":2836017,"kb_avail":374649188,"statfs":{"total":386547056640,"available":383640768512,"internally_reserved":0,"allocated":2105344,"data_stored":805553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2904081974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12647,"num_write_kb":-50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001309"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17259,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379419+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:21.379419+0000","last_peered":"2024-09-11T05:10:21.379419+0000","last_clean":"2024-09-11T05:10:21.379419+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:21.379419+0000","last_undegraded":"2024-09-11T05:10:21.379419+0000","last_fullsized":"2024-09-11T05:10:21.379419+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12919,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376575+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:21.376575+0000","last_peered":"2024-09-11T05:10:21.376575+0000","last_clean":"2024-09-11T05:10:21.376575+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:21.376575+0000","last_undegraded":"2024-09-11T05:10:21.376575+0000","last_fullsized":"2024-09-11T05:10:21.376575+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16765,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376550+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:21.376550+0000","last_peered":"2024-09-11T05:10:21.376550+0000","last_clean":"2024-09-11T05:10:21.376550+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:21.376550+0000","last_undegraded":"2024-09-11T05:10:21.376550+0000","last_fullsized":"2024-09-11T05:10:21.376550+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13987,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.375920+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:21.375920+0000","last_peered":"2024-09-11T05:10:21.375920+0000","last_clean":"2024-09-11T05:10:21.375920+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:21.375920+0000","last_undegraded":"2024-09-11T05:10:21.375920+0000","last_fullsized":"2024-09-11T05:10:21.375920+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17175,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379145+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:21.379145+0000","last_peered":"2024-09-11T05:10:21.379145+0000","last_clean":"2024-09-11T05:10:21.379145+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:21.379145+0000","last_undegraded":"2024-09-11T05:10:21.379145+0000","last_fullsized":"2024-09-11T05:10:21.379145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17402,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379352+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:21.379352+0000","last_peered":"2024-09-11T05:10:21.379352+0000","last_clean":"2024-09-11T05:10:21.379352+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:21.379352+0000","last_undegraded":"2024-09-11T05:10:21.379352+0000","last_fullsized":"2024-09-11T05:10:21.379352+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17517,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.382010+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:21.382010+0000","last_peered":"2024-09-11T05:10:21.382010+0000","last_clean":"2024-09-11T05:10:21.382010+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:21.382010+0000","last_undegraded":"2024-09-11T05:10:21.382010+0000","last_fullsized":"2024-09-11T05:10:21.382010+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17739,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376986+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:21.376986+0000","last_peered":"2024-09-11T05:10:21.376986+0000","last_clean":"2024-09-11T05:10:21.376986+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:21.376986+0000","last_undegraded":"2024-09-11T05:10:21.376986+0000","last_fullsized":"2024-09-11T05:10:21.376986+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17722,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379203+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:21.379203+0000","last_peered":"2024-09-11T05:10:21.379203+0000","last_clean":"2024-09-11T05:10:21.379203+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:21.379203+0000","last_undegraded":"2024-09-11T05:10:21.379203+0000","last_fullsized":"2024-09-11T05:10:21.379203+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17331,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376712+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:21.376712+0000","last_peered":"2024-09-11T05:10:21.376712+0000","last_clean":"2024-09-11T05:10:21.376712+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:21.376712+0000","last_undegraded":"2024-09-11T05:10:21.376712+0000","last_fullsized":"2024-09-11T05:10:21.376712+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17446,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:22.452891+0000","last_change":"2024-09-11T05:10:22.452891+0000","last_active":"2024-09-11T05:10:22.452891+0000","last_peered":"2024-09-11T05:10:22.452891+0000","last_clean":"2024-09-11T05:10:22.452891+0000","last_became_active":"2024-09-11T05:10:21.382743+0000","last_became_peered":"2024-09-11T05:10:21.382743+0000","last_unstale":"2024-09-11T05:10:22.452891+0000","last_undegraded":"2024-09-11T05:10:22.452891+0000","last_fullsized":"2024-09-11T05:10:22.452891+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12650","last_scrub_stamp":"2024-09-11T05:10:22.452806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:10:22.452806+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:52.349912+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17043,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377533+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:21.377533+0000","last_peered":"2024-09-11T05:10:21.377533+0000","last_clean":"2024-09-11T05:10:21.377533+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:21.377533+0000","last_undegraded":"2024-09-11T05:10:21.377533+0000","last_fullsized":"2024-09-11T05:10:21.377533+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376078+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:21.376078+0000","last_peered":"2024-09-11T05:10:21.376078+0000","last_clean":"2024-09-11T05:10:21.376078+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:21.376078+0000","last_undegraded":"2024-09-11T05:10:21.376078+0000","last_fullsized":"2024-09-11T05:10:21.376078+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12753,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376560+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:21.376560+0000","last_peered":"2024-09-11T05:10:21.376560+0000","last_clean":"2024-09-11T05:10:21.376560+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:21.376560+0000","last_undegraded":"2024-09-11T05:10:21.376560+0000","last_fullsized":"2024-09-11T05:10:21.376560+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12566,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376819+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:21.376819+0000","last_peered":"2024-09-11T05:10:21.376819+0000","last_clean":"2024-09-11T05:10:21.376819+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:21.376819+0000","last_undegraded":"2024-09-11T05:10:21.376819+0000","last_fullsized":"2024-09-11T05:10:21.376819+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12947,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377121+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:21.377121+0000","last_peered":"2024-09-11T05:10:21.377121+0000","last_clean":"2024-09-11T05:10:21.377121+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:21.377121+0000","last_undegraded":"2024-09-11T05:10:21.377121+0000","last_fullsized":"2024-09-11T05:10:21.377121+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744404,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627436,"statfs":{"total":96636764160,"available":95874494464,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721692,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650148,"statfs":{"total":96636764160,"available":95897751552,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649936,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93721904,"statfs":{"total":96636764160,"available":95971229696,"internally_reserved":0,"allocated":520192,"data_stored":199103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":664989394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:27.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:27.592 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:27.605 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-09-11T05:10:25.956535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838172,"kb_used_data":2056,"kb_used_omap":14,"kb_used_meta":2836017,"kb_avail":374649188,"statfs":{"total":386547056640,"available":383640768512,"internally_reserved":0,"allocated":2105344,"data_stored":805553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2904081974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12647,"num_write_kb":-50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001309"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17259,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379419+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:21.379419+0000","last_peered":"2024-09-11T05:10:21.379419+0000","last_clean":"2024-09-11T05:10:21.379419+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:21.379419+0000","last_undegraded":"2024-09-11T05:10:21.379419+0000","last_fullsized":"2024-09-11T05:10:21.379419+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12919,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376575+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:21.376575+0000","last_peered":"2024-09-11T05:10:21.376575+0000","last_clean":"2024-09-11T05:10:21.376575+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:21.376575+0000","last_undegraded":"2024-09-11T05:10:21.376575+0000","last_fullsized":"2024-09-11T05:10:21.376575+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16765,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376550+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:21.376550+0000","last_peered":"2024-09-11T05:10:21.376550+0000","last_clean":"2024-09-11T05:10:21.376550+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:21.376550+0000","last_undegraded":"2024-09-11T05:10:21.376550+0000","last_fullsized":"2024-09-11T05:10:21.376550+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13987,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.375920+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:21.375920+0000","last_peered":"2024-09-11T05:10:21.375920+0000","last_clean":"2024-09-11T05:10:21.375920+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:21.375920+0000","last_undegraded":"2024-09-11T05:10:21.375920+0000","last_fullsized":"2024-09-11T05:10:21.375920+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17175,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379145+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:21.379145+0000","last_peered":"2024-09-11T05:10:21.379145+0000","last_clean":"2024-09-11T05:10:21.379145+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:21.379145+0000","last_undegraded":"2024-09-11T05:10:21.379145+0000","last_fullsized":"2024-09-11T05:10:21.379145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17402,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379352+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:21.379352+0000","last_peered":"2024-09-11T05:10:21.379352+0000","last_clean":"2024-09-11T05:10:21.379352+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:21.379352+0000","last_undegraded":"2024-09-11T05:10:21.379352+0000","last_fullsized":"2024-09-11T05:10:21.379352+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17517,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.382010+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:21.382010+0000","last_peered":"2024-09-11T05:10:21.382010+0000","last_clean":"2024-09-11T05:10:21.382010+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:21.382010+0000","last_undegraded":"2024-09-11T05:10:21.382010+0000","last_fullsized":"2024-09-11T05:10:21.382010+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17739,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376986+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:21.376986+0000","last_peered":"2024-09-11T05:10:21.376986+0000","last_clean":"2024-09-11T05:10:21.376986+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:21.376986+0000","last_undegraded":"2024-09-11T05:10:21.376986+0000","last_fullsized":"2024-09-11T05:10:21.376986+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17722,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379203+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:21.379203+0000","last_peered":"2024-09-11T05:10:21.379203+0000","last_clean":"2024-09-11T05:10:21.379203+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:21.379203+0000","last_undegraded":"2024-09-11T05:10:21.379203+0000","last_fullsized":"2024-09-11T05:10:21.379203+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17331,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376712+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:21.376712+0000","last_peered":"2024-09-11T05:10:21.376712+0000","last_clean":"2024-09-11T05:10:21.376712+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:21.376712+0000","last_undegraded":"2024-09-11T05:10:21.376712+0000","last_fullsized":"2024-09-11T05:10:21.376712+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17446,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:22.452891+0000","last_change":"2024-09-11T05:10:22.452891+0000","last_active":"2024-09-11T05:10:22.452891+0000","last_peered":"2024-09-11T05:10:22.452891+0000","last_clean":"2024-09-11T05:10:22.452891+0000","last_became_active":"2024-09-11T05:10:21.382743+0000","last_became_peered":"2024-09-11T05:10:21.382743+0000","last_unstale":"2024-09-11T05:10:22.452891+0000","last_undegraded":"2024-09-11T05:10:22.452891+0000","last_fullsized":"2024-09-11T05:10:22.452891+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12650","last_scrub_stamp":"2024-09-11T05:10:22.452806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:10:22.452806+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:52.349912+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17043,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377533+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:21.377533+0000","last_peered":"2024-09-11T05:10:21.377533+0000","last_clean":"2024-09-11T05:10:21.377533+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:21.377533+0000","last_undegraded":"2024-09-11T05:10:21.377533+0000","last_fullsized":"2024-09-11T05:10:21.377533+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376078+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:21.376078+0000","last_peered":"2024-09-11T05:10:21.376078+0000","last_clean":"2024-09-11T05:10:21.376078+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:21.376078+0000","last_undegraded":"2024-09-11T05:10:21.376078+0000","last_fullsized":"2024-09-11T05:10:21.376078+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12753,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376560+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:21.376560+0000","last_peered":"2024-09-11T05:10:21.376560+0000","last_clean":"2024-09-11T05:10:21.376560+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:21.376560+0000","last_undegraded":"2024-09-11T05:10:21.376560+0000","last_fullsized":"2024-09-11T05:10:21.376560+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12566,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376819+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:21.376819+0000","last_peered":"2024-09-11T05:10:21.376819+0000","last_clean":"2024-09-11T05:10:21.376819+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:21.376819+0000","last_undegraded":"2024-09-11T05:10:21.376819+0000","last_fullsized":"2024-09-11T05:10:21.376819+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12947,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377121+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:21.377121+0000","last_peered":"2024-09-11T05:10:21.377121+0000","last_clean":"2024-09-11T05:10:21.377121+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:21.377121+0000","last_undegraded":"2024-09-11T05:10:21.377121+0000","last_fullsized":"2024-09-11T05:10:21.377121+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744404,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627436,"statfs":{"total":96636764160,"available":95874494464,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721692,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650148,"statfs":{"total":96636764160,"available":95897751552,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649936,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93721904,"statfs":{"total":96636764160,"available":95971229696,"internally_reserved":0,"allocated":520192,"data_stored":199103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":664989394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:27.606 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T05:10:27.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:27.901 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:27.912 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-09-11T05:10:25.956535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838172,"kb_used_data":2056,"kb_used_omap":14,"kb_used_meta":2836017,"kb_avail":374649188,"statfs":{"total":386547056640,"available":383640768512,"internally_reserved":0,"allocated":2105344,"data_stored":805553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2904081974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12647,"num_write_kb":-50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001309"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17259,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379419+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:21.379419+0000","last_peered":"2024-09-11T05:10:21.379419+0000","last_clean":"2024-09-11T05:10:21.379419+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:21.379419+0000","last_undegraded":"2024-09-11T05:10:21.379419+0000","last_fullsized":"2024-09-11T05:10:21.379419+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12919,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376575+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:21.376575+0000","last_peered":"2024-09-11T05:10:21.376575+0000","last_clean":"2024-09-11T05:10:21.376575+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:21.376575+0000","last_undegraded":"2024-09-11T05:10:21.376575+0000","last_fullsized":"2024-09-11T05:10:21.376575+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16765,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376550+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:21.376550+0000","last_peered":"2024-09-11T05:10:21.376550+0000","last_clean":"2024-09-11T05:10:21.376550+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:21.376550+0000","last_undegraded":"2024-09-11T05:10:21.376550+0000","last_fullsized":"2024-09-11T05:10:21.376550+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13987,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.375920+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:21.375920+0000","last_peered":"2024-09-11T05:10:21.375920+0000","last_clean":"2024-09-11T05:10:21.375920+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:21.375920+0000","last_undegraded":"2024-09-11T05:10:21.375920+0000","last_fullsized":"2024-09-11T05:10:21.375920+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17175,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379145+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:21.379145+0000","last_peered":"2024-09-11T05:10:21.379145+0000","last_clean":"2024-09-11T05:10:21.379145+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:21.379145+0000","last_undegraded":"2024-09-11T05:10:21.379145+0000","last_fullsized":"2024-09-11T05:10:21.379145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17402,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379352+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:21.379352+0000","last_peered":"2024-09-11T05:10:21.379352+0000","last_clean":"2024-09-11T05:10:21.379352+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:21.379352+0000","last_undegraded":"2024-09-11T05:10:21.379352+0000","last_fullsized":"2024-09-11T05:10:21.379352+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17517,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.382010+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:21.382010+0000","last_peered":"2024-09-11T05:10:21.382010+0000","last_clean":"2024-09-11T05:10:21.382010+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:21.382010+0000","last_undegraded":"2024-09-11T05:10:21.382010+0000","last_fullsized":"2024-09-11T05:10:21.382010+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17739,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376986+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:21.376986+0000","last_peered":"2024-09-11T05:10:21.376986+0000","last_clean":"2024-09-11T05:10:21.376986+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:21.376986+0000","last_undegraded":"2024-09-11T05:10:21.376986+0000","last_fullsized":"2024-09-11T05:10:21.376986+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17722,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379203+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:21.379203+0000","last_peered":"2024-09-11T05:10:21.379203+0000","last_clean":"2024-09-11T05:10:21.379203+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:21.379203+0000","last_undegraded":"2024-09-11T05:10:21.379203+0000","last_fullsized":"2024-09-11T05:10:21.379203+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17331,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376712+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:21.376712+0000","last_peered":"2024-09-11T05:10:21.376712+0000","last_clean":"2024-09-11T05:10:21.376712+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:21.376712+0000","last_undegraded":"2024-09-11T05:10:21.376712+0000","last_fullsized":"2024-09-11T05:10:21.376712+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17446,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:22.452891+0000","last_change":"2024-09-11T05:10:22.452891+0000","last_active":"2024-09-11T05:10:22.452891+0000","last_peered":"2024-09-11T05:10:22.452891+0000","last_clean":"2024-09-11T05:10:22.452891+0000","last_became_active":"2024-09-11T05:10:21.382743+0000","last_became_peered":"2024-09-11T05:10:21.382743+0000","last_unstale":"2024-09-11T05:10:22.452891+0000","last_undegraded":"2024-09-11T05:10:22.452891+0000","last_fullsized":"2024-09-11T05:10:22.452891+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12650","last_scrub_stamp":"2024-09-11T05:10:22.452806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:10:22.452806+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:52.349912+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17043,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377533+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:21.377533+0000","last_peered":"2024-09-11T05:10:21.377533+0000","last_clean":"2024-09-11T05:10:21.377533+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:21.377533+0000","last_undegraded":"2024-09-11T05:10:21.377533+0000","last_fullsized":"2024-09-11T05:10:21.377533+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376078+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:21.376078+0000","last_peered":"2024-09-11T05:10:21.376078+0000","last_clean":"2024-09-11T05:10:21.376078+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:21.376078+0000","last_undegraded":"2024-09-11T05:10:21.376078+0000","last_fullsized":"2024-09-11T05:10:21.376078+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12753,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376560+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:21.376560+0000","last_peered":"2024-09-11T05:10:21.376560+0000","last_clean":"2024-09-11T05:10:21.376560+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:21.376560+0000","last_undegraded":"2024-09-11T05:10:21.376560+0000","last_fullsized":"2024-09-11T05:10:21.376560+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12566,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376819+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:21.376819+0000","last_peered":"2024-09-11T05:10:21.376819+0000","last_clean":"2024-09-11T05:10:21.376819+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:21.376819+0000","last_undegraded":"2024-09-11T05:10:21.376819+0000","last_fullsized":"2024-09-11T05:10:21.376819+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12947,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377121+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:21.377121+0000","last_peered":"2024-09-11T05:10:21.377121+0000","last_clean":"2024-09-11T05:10:21.377121+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:21.377121+0000","last_undegraded":"2024-09-11T05:10:21.377121+0000","last_fullsized":"2024-09-11T05:10:21.377121+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744404,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627436,"statfs":{"total":96636764160,"available":95874494464,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721692,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650148,"statfs":{"total":96636764160,"available":95897751552,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649936,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93721904,"statfs":{"total":96636764160,"available":95971229696,"internally_reserved":0,"allocated":520192,"data_stored":199103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":664989394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:27.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:10:28.214 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:28.214 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":63,"fsid":"843fb9a6-9086-48d4-b349-4200e697289c","created":"2024-09-11T04:54:33.004393+0000","modified":"2024-09-11T05:10:21.372829+0000","last_up_change":"2024-09-11T04:56:18.205049+0000","last_in_change":"2024-09-11T04:54:34.575381+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-11T04:54:55.691815+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"0a0d5d52-301d-428c-877b-eefc61f019f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":49,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":448159694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":448159694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":448159694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":448159694}]},"public_addr":"172.21.15.31:6800/448159694","cluster_addr":"172.21.15.31:6801/448159694","heartbeat_back_addr":"172.21.15.31:6803/448159694","heartbeat_front_addr":"172.21.15.31:6802/448159694","state":["exists","up"]},{"osd":1,"uuid":"83f13be3-d367-46d4-8128-98883066e4db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":56,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":815718084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":815718084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":815718084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":815718084}]},"public_addr":"172.21.15.31:6812/815718084","cluster_addr":"172.21.15.31:6813/815718084","heartbeat_back_addr":"172.21.15.31:6815/815718084","heartbeat_front_addr":"172.21.15.31:6814/815718084","state":["exists","up"]},{"osd":2,"uuid":"617490bf-6727-44cc-92d8-5215e439c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2027023681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2027023681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2027023681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2027023681}]},"public_addr":"172.21.15.31:6808/2027023681","cluster_addr":"172.21.15.31:6809/2027023681","heartbeat_back_addr":"172.21.15.31:6811/2027023681","heartbeat_front_addr":"172.21.15.31:6810/2027023681","state":["exists","up"]},{"osd":3,"uuid":"39dfc5a1-9332-4ad1-b51e-d8502c155301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":62,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2434669726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2434669726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":2434669726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2434669726}]},"public_addr":"172.21.15.31:6804/2434669726","cluster_addr":"172.21.15.31:6805/2434669726","heartbeat_back_addr":"172.21.15.31:6807/2434669726","heartbeat_front_addr":"172.21.15.31:6806/2434669726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.833079+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.663344+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:40.858324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:54:41.136761+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:10:29.228 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T05:10:29.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:10:29.370 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T05:10:29.370 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:10:29.370 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T05:10:29.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T05:10:29.681 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T05:10:29.694 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T05:10:29.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:10:29.820 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T05:10:29.821 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:10:29.821 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T05:10:29.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T05:10:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T05:10:30.147 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T05:10:30.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:10:30.276 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T05:10:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:10:30.276 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T05:10:30.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T05:10:30.585 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T05:10:30.598 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T05:10:30.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:10:30.724 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T05:10:30.724 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:10:30.724 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T05:10:30.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T05:10:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T05:10:31.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:31.352 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:31.364 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-09-11T05:10:29.957165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838180,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2836017,"kb_avail":374649180,"statfs":{"total":386547056640,"available":383640760320,"internally_reserved":0,"allocated":2113536,"data_stored":808600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2904081974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20208,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12647,"num_write_kb":-50596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.583456"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17259,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379419+0000","last_change":"2024-09-11T05:10:09.011487+0000","last_active":"2024-09-11T05:10:21.379419+0000","last_peered":"2024-09-11T05:10:21.379419+0000","last_clean":"2024-09-11T05:10:21.379419+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:21.379419+0000","last_undegraded":"2024-09-11T05:10:21.379419+0000","last_fullsized":"2024-09-11T05:10:21.379419+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:59:07.289251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12919,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376575+0000","last_change":"2024-09-11T05:08:22.475708+0000","last_active":"2024-09-11T05:10:21.376575+0000","last_peered":"2024-09-11T05:10:21.376575+0000","last_clean":"2024-09-11T05:10:21.376575+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:21.376575+0000","last_undegraded":"2024-09-11T05:10:21.376575+0000","last_fullsized":"2024-09-11T05:10:21.376575+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:08:22.475627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:22.475627+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:13.866865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16765,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376550+0000","last_change":"2024-09-11T05:09:23.176199+0000","last_active":"2024-09-11T05:10:21.376550+0000","last_peered":"2024-09-11T05:10:21.376550+0000","last_clean":"2024-09-11T05:10:21.376550+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:21.376550+0000","last_undegraded":"2024-09-11T05:10:21.376550+0000","last_fullsized":"2024-09-11T05:10:21.376550+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:09:23.176083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:23.176083+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:23:30.805205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13987,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.375920+0000","last_change":"2024-09-11T05:07:59.779639+0000","last_active":"2024-09-11T05:10:21.375920+0000","last_peered":"2024-09-11T05:10:21.375920+0000","last_clean":"2024-09-11T05:10:21.375920+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:21.375920+0000","last_undegraded":"2024-09-11T05:10:21.375920+0000","last_fullsized":"2024-09-11T05:10:21.375920+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:07:59.779574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:07:59.779574+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:17.414141+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17175,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379145+0000","last_change":"2024-09-11T05:08:11.278519+0000","last_active":"2024-09-11T05:10:21.379145+0000","last_peered":"2024-09-11T05:10:21.379145+0000","last_clean":"2024-09-11T05:10:21.379145+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:21.379145+0000","last_undegraded":"2024-09-11T05:10:21.379145+0000","last_fullsized":"2024-09-11T05:10:21.379145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:08:11.278438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:11.278438+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:27.126153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17402,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379352+0000","last_change":"2024-09-11T05:09:33.322603+0000","last_active":"2024-09-11T05:10:21.379352+0000","last_peered":"2024-09-11T05:10:21.379352+0000","last_clean":"2024-09-11T05:10:21.379352+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:21.379352+0000","last_undegraded":"2024-09-11T05:10:21.379352+0000","last_fullsized":"2024-09-11T05:10:21.379352+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:54:56.683804+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:53:15.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17517,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.382010+0000","last_change":"2024-09-11T05:08:46.403995+0000","last_active":"2024-09-11T05:10:21.382010+0000","last_peered":"2024-09-11T05:10:21.382010+0000","last_clean":"2024-09-11T05:10:21.382010+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:21.382010+0000","last_undegraded":"2024-09-11T05:10:21.382010+0000","last_fullsized":"2024-09-11T05:10:21.382010+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:08:46.403498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:46.403498+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:59.410115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17739,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376986+0000","last_change":"2024-09-11T05:03:06.390095+0000","last_active":"2024-09-11T05:10:21.376986+0000","last_peered":"2024-09-11T05:10:21.376986+0000","last_clean":"2024-09-11T05:10:21.376986+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:21.376986+0000","last_undegraded":"2024-09-11T05:10:21.376986+0000","last_fullsized":"2024-09-11T05:10:21.376986+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9328","last_scrub_stamp":"2024-09-11T05:03:06.389816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:03:06.389816+0000","objects_scrubbed":3533,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:46.611563+0000","scrub_duration":7213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17722,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.379203+0000","last_change":"2024-09-11T05:09:57.409213+0000","last_active":"2024-09-11T05:10:21.379203+0000","last_peered":"2024-09-11T05:10:21.379203+0000","last_clean":"2024-09-11T05:10:21.379203+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:21.379203+0000","last_undegraded":"2024-09-11T05:10:21.379203+0000","last_fullsized":"2024-09-11T05:10:21.379203+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:09:57.409121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:57.409121+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:23.084883+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17331,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376712+0000","last_change":"2024-09-11T05:08:34.206929+0000","last_active":"2024-09-11T05:10:21.376712+0000","last_peered":"2024-09-11T05:10:21.376712+0000","last_clean":"2024-09-11T05:10:21.376712+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:21.376712+0000","last_undegraded":"2024-09-11T05:10:21.376712+0000","last_fullsized":"2024-09-11T05:10:21.376712+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:08:34.206853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:34.206853+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:50.313795+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17446,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:22.452891+0000","last_change":"2024-09-11T05:10:22.452891+0000","last_active":"2024-09-11T05:10:22.452891+0000","last_peered":"2024-09-11T05:10:22.452891+0000","last_clean":"2024-09-11T05:10:22.452891+0000","last_became_active":"2024-09-11T05:10:21.382743+0000","last_became_peered":"2024-09-11T05:10:21.382743+0000","last_unstale":"2024-09-11T05:10:22.452891+0000","last_undegraded":"2024-09-11T05:10:22.452891+0000","last_fullsized":"2024-09-11T05:10:22.452891+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12650","last_scrub_stamp":"2024-09-11T05:10:22.452806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:10:22.452806+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:52.349912+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17044,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377587+0000","last_change":"2024-09-11T04:59:44.288949+0000","last_active":"2024-09-11T05:10:21.377587+0000","last_peered":"2024-09-11T05:10:21.377587+0000","last_clean":"2024-09-11T05:10:21.377587+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:21.377587+0000","last_undegraded":"2024-09-11T05:10:21.377587+0000","last_fullsized":"2024-09-11T05:10:21.377587+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'6193","last_scrub_stamp":"2024-09-11T04:59:44.288783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T04:59:44.288783+0000","objects_scrubbed":6188,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:24.113071+0000","scrub_duration":3257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376078+0000","last_change":"2024-09-11T05:06:40.458020+0000","last_active":"2024-09-11T05:10:21.376078+0000","last_peered":"2024-09-11T05:10:21.376078+0000","last_clean":"2024-09-11T05:10:21.376078+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:21.376078+0000","last_undegraded":"2024-09-11T05:10:21.376078+0000","last_fullsized":"2024-09-11T05:10:21.376078+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11855","last_scrub_stamp":"2024-09-11T05:06:40.457688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:06:40.457688+0000","objects_scrubbed":869,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:25.040306+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12753,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376560+0000","last_change":"2024-09-11T05:08:58.817214+0000","last_active":"2024-09-11T05:10:21.376560+0000","last_peered":"2024-09-11T05:10:21.376560+0000","last_clean":"2024-09-11T05:10:21.376560+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:21.376560+0000","last_undegraded":"2024-09-11T05:10:21.376560+0000","last_fullsized":"2024-09-11T05:10:21.376560+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:08:58.817114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:08:58.817114+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:29.619161+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12566,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.376819+0000","last_change":"2024-09-11T05:09:44.679526+0000","last_active":"2024-09-11T05:10:21.376819+0000","last_peered":"2024-09-11T05:10:21.376819+0000","last_clean":"2024-09-11T05:10:21.376819+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:21.376819+0000","last_undegraded":"2024-09-11T05:10:21.376819+0000","last_fullsized":"2024-09-11T05:10:21.376819+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:09:44.679458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:44.679458+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:39:25.609015+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12947,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:21.377121+0000","last_change":"2024-09-11T05:09:10.916179+0000","last_active":"2024-09-11T05:10:21.377121+0000","last_peered":"2024-09-11T05:10:21.377121+0000","last_clean":"2024-09-11T05:10:21.377121+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:21.377121+0000","last_undegraded":"2024-09-11T05:10:21.377121+0000","last_fullsized":"2024-09-11T05:10:21.377121+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:09:10.916106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:54:56.683804+0000","last_clean_scrub_stamp":"2024-09-11T05:09:10.916106+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:13.977801+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744404,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627436,"statfs":{"total":96636764160,"available":95874494464,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721692,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650148,"statfs":{"total":96636764160,"available":95897751552,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649944,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93721896,"statfs":{"total":96636764160,"available":95971221504,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":664989394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:31.368 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-09-11T04:54:56.683804+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=54, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.369 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-09-11T05:08:22.475627+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=22, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.369 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-09-11T05:09:23.176083+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=9, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.369 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T05:07:59.779574+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=59, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.369 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-11T05:08:11.278438+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.369 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-09-11T04:54:56.683804+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=54, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.370 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-11T05:08:46.403498+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=46, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.370 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-11T05:03:06.389816+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=3, tm_sec=6, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.370 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-11T05:09:57.409121+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=9, tm_sec=57, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.370 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-11T05:08:34.206853+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.370 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-11T05:10:22.452806+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=22, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.370 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-11T04:59:44.288783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=59, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.370 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-09-11T05:06:40.457688+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=6, tm_sec=40, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.371 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-09-11T05:08:58.817114+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=58, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.371 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-09-11T05:09:44.679458+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=9, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.371 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-09-11T05:09:10.916106+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=10, tm_sec=28, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:10:31.371 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T05:10:51.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:10:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T05:10:51.698 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T05:10:51.711 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":543,"stamp":"2024-09-11T05:10:49.960770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2838160,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2836017,"kb_avail":374649200,"statfs":{"total":386547056640,"available":383640780800,"internally_reserved":0,"allocated":2113536,"data_stored":808600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14794,"internal_metadata":2904081974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002288"},"pg_stats":[{"pgid":"1.f","version":"37'12514","reported_seq":17267,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:35.377523+0000","last_change":"2024-09-11T05:10:35.377523+0000","last_active":"2024-09-11T05:10:35.377523+0000","last_peered":"2024-09-11T05:10:35.377523+0000","last_clean":"2024-09-11T05:10:35.377523+0000","last_became_active":"2024-09-11T05:10:08.935445+0000","last_became_peered":"2024-09-11T05:10:08.935445+0000","last_unstale":"2024-09-11T05:10:35.377523+0000","last_undegraded":"2024-09-11T05:10:35.377523+0000","last_fullsized":"2024-09-11T05:10:35.377523+0000","mapping_epoch":60,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12514","last_scrub_stamp":"2024-09-11T05:10:35.377414+0000","last_deep_scrub":"37'12514","last_deep_scrub_stamp":"2024-09-11T05:10:35.377414+0000","last_clean_scrub_stamp":"2024-09-11T05:10:35.377414+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:13.186515+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":50056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12736","reported_seq":12928,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:37.175983+0000","last_change":"2024-09-11T05:10:37.175983+0000","last_active":"2024-09-11T05:10:37.175983+0000","last_peered":"2024-09-11T05:10:37.175983+0000","last_clean":"2024-09-11T05:10:37.175983+0000","last_became_active":"2024-09-11T05:08:22.266322+0000","last_became_peered":"2024-09-11T05:08:22.266322+0000","last_unstale":"2024-09-11T05:10:37.175983+0000","last_undegraded":"2024-09-11T05:10:37.175983+0000","last_fullsized":"2024-09-11T05:10:37.175983+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12736","last_scrub_stamp":"2024-09-11T05:10:37.175875+0000","last_deep_scrub":"37'12736","last_deep_scrub_stamp":"2024-09-11T05:10:37.175875+0000","last_clean_scrub_stamp":"2024-09-11T05:10:37.175875+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:30:50.710935+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12160","reported_seq":16773,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:32.120557+0000","last_change":"2024-09-11T05:10:32.120557+0000","last_active":"2024-09-11T05:10:32.120557+0000","last_peered":"2024-09-11T05:10:32.120557+0000","last_clean":"2024-09-11T05:10:32.120557+0000","last_became_active":"2024-09-11T05:09:22.203629+0000","last_became_peered":"2024-09-11T05:09:22.203629+0000","last_unstale":"2024-09-11T05:10:32.120557+0000","last_undegraded":"2024-09-11T05:10:32.120557+0000","last_fullsized":"2024-09-11T05:10:32.120557+0000","mapping_epoch":52,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12160","last_scrub_stamp":"2024-09-11T05:10:32.120448+0000","last_deep_scrub":"37'12160","last_deep_scrub_stamp":"2024-09-11T05:10:32.120448+0000","last_clean_scrub_stamp":"2024-09-11T05:10:32.120448+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:41:41.630470+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12414","reported_seq":13995,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:35.235394+0000","last_change":"2024-09-11T05:10:35.235394+0000","last_active":"2024-09-11T05:10:35.235394+0000","last_peered":"2024-09-11T05:10:35.235394+0000","last_clean":"2024-09-11T05:10:35.235394+0000","last_became_active":"2024-09-11T05:07:58.925201+0000","last_became_peered":"2024-09-11T05:07:58.925201+0000","last_unstale":"2024-09-11T05:10:35.235394+0000","last_undegraded":"2024-09-11T05:10:35.235394+0000","last_fullsized":"2024-09-11T05:10:35.235394+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12414","last_scrub_stamp":"2024-09-11T05:10:35.235283+0000","last_deep_scrub":"37'12414","last_deep_scrub_stamp":"2024-09-11T05:10:35.235283+0000","last_clean_scrub_stamp":"2024-09-11T05:10:35.235283+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:52:55.681784+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12529","reported_seq":17183,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:32.371681+0000","last_change":"2024-09-11T05:10:32.371681+0000","last_active":"2024-09-11T05:10:32.371681+0000","last_peered":"2024-09-11T05:10:32.371681+0000","last_clean":"2024-09-11T05:10:32.371681+0000","last_became_active":"2024-09-11T05:08:10.232355+0000","last_became_peered":"2024-09-11T05:08:10.232355+0000","last_unstale":"2024-09-11T05:10:32.371681+0000","last_undegraded":"2024-09-11T05:10:32.371681+0000","last_fullsized":"2024-09-11T05:10:32.371681+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12529","last_scrub_stamp":"2024-09-11T05:10:32.371544+0000","last_deep_scrub":"37'12529","last_deep_scrub_stamp":"2024-09-11T05:10:32.371544+0000","last_clean_scrub_stamp":"2024-09-11T05:10:32.371544+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:50.346619+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12640","reported_seq":17410,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:33.330810+0000","last_change":"2024-09-11T05:10:33.330810+0000","last_active":"2024-09-11T05:10:33.330810+0000","last_peered":"2024-09-11T05:10:33.330810+0000","last_clean":"2024-09-11T05:10:33.330810+0000","last_became_active":"2024-09-11T05:09:33.240373+0000","last_became_peered":"2024-09-11T05:09:33.240373+0000","last_unstale":"2024-09-11T05:10:33.330810+0000","last_undegraded":"2024-09-11T05:10:33.330810+0000","last_fullsized":"2024-09-11T05:10:33.330810+0000","mapping_epoch":54,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12640","last_scrub_stamp":"2024-09-11T05:10:33.330706+0000","last_deep_scrub":"37'12640","last_deep_scrub_stamp":"2024-09-11T05:10:33.330706+0000","last_clean_scrub_stamp":"2024-09-11T05:10:33.330706+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:13:42.074205+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12728","reported_seq":17525,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:30.136507+0000","last_change":"2024-09-11T05:10:30.136507+0000","last_active":"2024-09-11T05:10:30.136507+0000","last_peered":"2024-09-11T05:10:30.136507+0000","last_clean":"2024-09-11T05:10:30.136507+0000","last_became_active":"2024-09-11T05:08:46.034112+0000","last_became_peered":"2024-09-11T05:08:46.034112+0000","last_unstale":"2024-09-11T05:10:30.136507+0000","last_undegraded":"2024-09-11T05:10:30.136507+0000","last_fullsized":"2024-09-11T05:10:30.136507+0000","mapping_epoch":46,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12728","last_scrub_stamp":"2024-09-11T05:10:30.136404+0000","last_deep_scrub":"37'12728","last_deep_scrub_stamp":"2024-09-11T05:10:30.136404+0000","last_clean_scrub_stamp":"2024-09-11T05:10:30.136404+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:51.536657+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12823","reported_seq":17747,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:30.265767+0000","last_change":"2024-09-11T05:10:30.265767+0000","last_active":"2024-09-11T05:10:30.265767+0000","last_peered":"2024-09-11T05:10:30.265767+0000","last_clean":"2024-09-11T05:10:30.265767+0000","last_became_active":"2024-09-11T05:02:58.942760+0000","last_became_peered":"2024-09-11T05:02:58.942760+0000","last_unstale":"2024-09-11T05:10:30.265767+0000","last_undegraded":"2024-09-11T05:10:30.265767+0000","last_fullsized":"2024-09-11T05:10:30.265767+0000","mapping_epoch":34,"log_start":"31'2800","ondisk_log_start":"31'2800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12823","last_scrub_stamp":"2024-09-11T05:10:30.265680+0000","last_deep_scrub":"37'12823","last_deep_scrub_stamp":"2024-09-11T05:10:30.265680+0000","last_clean_scrub_stamp":"2024-09-11T05:10:30.265680+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:07:08.593708+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12784","reported_seq":17730,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:34.350573+0000","last_change":"2024-09-11T05:10:34.350573+0000","last_active":"2024-09-11T05:10:34.350573+0000","last_peered":"2024-09-11T05:10:34.350573+0000","last_clean":"2024-09-11T05:10:34.350573+0000","last_became_active":"2024-09-11T05:09:57.313859+0000","last_became_peered":"2024-09-11T05:09:57.313859+0000","last_unstale":"2024-09-11T05:10:34.350573+0000","last_undegraded":"2024-09-11T05:10:34.350573+0000","last_fullsized":"2024-09-11T05:10:34.350573+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12784","last_scrub_stamp":"2024-09-11T05:10:34.350471+0000","last_deep_scrub":"37'12784","last_deep_scrub_stamp":"2024-09-11T05:10:34.350471+0000","last_clean_scrub_stamp":"2024-09-11T05:10:34.350471+0000","objects_scrubbed":1,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:49:17.742449+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":17339,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:31.107809+0000","last_change":"2024-09-11T05:10:31.107809+0000","last_active":"2024-09-11T05:10:31.107809+0000","last_peered":"2024-09-11T05:10:31.107809+0000","last_clean":"2024-09-11T05:10:31.107809+0000","last_became_active":"2024-09-11T05:08:33.921078+0000","last_became_peered":"2024-09-11T05:08:33.921078+0000","last_unstale":"2024-09-11T05:10:31.107809+0000","last_undegraded":"2024-09-11T05:10:31.107809+0000","last_fullsized":"2024-09-11T05:10:31.107809+0000","mapping_epoch":44,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12526","last_scrub_stamp":"2024-09-11T05:10:31.107722+0000","last_deep_scrub":"37'12526","last_deep_scrub_stamp":"2024-09-11T05:10:31.107722+0000","last_clean_scrub_stamp":"2024-09-11T05:10:31.107722+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:04:40.441708+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12650","reported_seq":17454,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:31.342293+0000","last_change":"2024-09-11T05:10:31.342293+0000","last_active":"2024-09-11T05:10:31.342293+0000","last_peered":"2024-09-11T05:10:31.342293+0000","last_clean":"2024-09-11T05:10:31.342293+0000","last_became_active":"2024-09-11T05:10:21.382743+0000","last_became_peered":"2024-09-11T05:10:21.382743+0000","last_unstale":"2024-09-11T05:10:31.342293+0000","last_undegraded":"2024-09-11T05:10:31.342293+0000","last_fullsized":"2024-09-11T05:10:31.342293+0000","mapping_epoch":62,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12650","last_scrub_stamp":"2024-09-11T05:10:31.342207+0000","last_deep_scrub":"37'12650","last_deep_scrub_stamp":"2024-09-11T05:10:31.342207+0000","last_clean_scrub_stamp":"2024-09-11T05:10:31.342207+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:14.847530+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12384","reported_seq":17052,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:31.277674+0000","last_change":"2024-09-11T05:10:31.277674+0000","last_active":"2024-09-11T05:10:31.277674+0000","last_peered":"2024-09-11T05:10:31.277674+0000","last_clean":"2024-09-11T05:10:31.277674+0000","last_became_active":"2024-09-11T04:59:40.053452+0000","last_became_peered":"2024-09-11T04:59:40.053452+0000","last_unstale":"2024-09-11T05:10:31.277674+0000","last_undegraded":"2024-09-11T05:10:31.277674+0000","last_fullsized":"2024-09-11T05:10:31.277674+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12384","last_scrub_stamp":"2024-09-11T05:10:31.277524+0000","last_deep_scrub":"37'12384","last_deep_scrub_stamp":"2024-09-11T05:10:31.277524+0000","last_clean_scrub_stamp":"2024-09-11T05:10:31.277524+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:26:42.575636+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12722","reported_seq":17565,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:36.207473+0000","last_change":"2024-09-11T05:10:36.207473+0000","last_active":"2024-09-11T05:10:36.207473+0000","last_peered":"2024-09-11T05:10:36.207473+0000","last_clean":"2024-09-11T05:10:36.207473+0000","last_became_active":"2024-09-11T05:06:38.153760+0000","last_became_peered":"2024-09-11T05:06:38.153760+0000","last_unstale":"2024-09-11T05:10:36.207473+0000","last_undegraded":"2024-09-11T05:10:36.207473+0000","last_fullsized":"2024-09-11T05:10:36.207473+0000","mapping_epoch":36,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12722","last_scrub_stamp":"2024-09-11T05:10:36.207340+0000","last_deep_scrub":"37'12722","last_deep_scrub_stamp":"2024-09-11T05:10:36.207340+0000","last_clean_scrub_stamp":"2024-09-11T05:10:36.207340+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:34:43.681693+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12590","reported_seq":12761,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:33.261152+0000","last_change":"2024-09-11T05:10:33.261152+0000","last_active":"2024-09-11T05:10:33.261152+0000","last_peered":"2024-09-11T05:10:33.261152+0000","last_clean":"2024-09-11T05:10:33.261152+0000","last_became_active":"2024-09-11T05:08:58.077563+0000","last_became_peered":"2024-09-11T05:08:58.077563+0000","last_unstale":"2024-09-11T05:10:33.261152+0000","last_undegraded":"2024-09-11T05:10:33.261152+0000","last_fullsized":"2024-09-11T05:10:33.261152+0000","mapping_epoch":48,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12590","last_scrub_stamp":"2024-09-11T05:10:33.260205+0000","last_deep_scrub":"37'12590","last_deep_scrub_stamp":"2024-09-11T05:10:33.260205+0000","last_clean_scrub_stamp":"2024-09-11T05:10:33.260205+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:52:26.811999+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12407","reported_seq":12574,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:32.299979+0000","last_change":"2024-09-11T05:10:32.299979+0000","last_active":"2024-09-11T05:10:32.299979+0000","last_peered":"2024-09-11T05:10:32.299979+0000","last_clean":"2024-09-11T05:10:32.299979+0000","last_became_active":"2024-09-11T05:09:44.270765+0000","last_became_peered":"2024-09-11T05:09:44.270765+0000","last_unstale":"2024-09-11T05:10:32.299979+0000","last_undegraded":"2024-09-11T05:10:32.299979+0000","last_fullsized":"2024-09-11T05:10:32.299979+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12407","last_scrub_stamp":"2024-09-11T05:10:32.299818+0000","last_deep_scrub":"37'12407","last_deep_scrub_stamp":"2024-09-11T05:10:32.299818+0000","last_clean_scrub_stamp":"2024-09-11T05:10:32.299818+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:46:07.207463+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12775","reported_seq":12955,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T05:10:34.236704+0000","last_change":"2024-09-11T05:10:34.236704+0000","last_active":"2024-09-11T05:10:34.236704+0000","last_peered":"2024-09-11T05:10:34.236704+0000","last_clean":"2024-09-11T05:10:34.236704+0000","last_became_active":"2024-09-11T05:09:10.163077+0000","last_became_peered":"2024-09-11T05:09:10.163077+0000","last_unstale":"2024-09-11T05:10:34.236704+0000","last_undegraded":"2024-09-11T05:10:34.236704+0000","last_fullsized":"2024-09-11T05:10:34.236704+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12775","last_scrub_stamp":"2024-09-11T05:10:34.236546+0000","last_deep_scrub":"37'12775","last_deep_scrub_stamp":"2024-09-11T05:10:34.236546+0000","last_clean_scrub_stamp":"2024-09-11T05:10:34.236546+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:19:54.606060+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6736,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35750,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160682,"ondisk_log_size":160682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744404,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":743867,"kb_avail":93627436,"statfs":{"total":96636764160,"available":95874494464,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":761720528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721684,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93650156,"statfs":{"total":96636764160,"available":95897759744,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649940,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93721900,"statfs":{"total":96636764160,"available":95971225600,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":664989394},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722132,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649708,"statfs":{"total":96636764160,"available":95897300992,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738913999},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:10:51.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T05:10:52.027 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T05:10:52.027 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T05:10:52.028 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T05:10:52.028 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:10:57.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-11T05:10:58.129 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T05:10:58.129 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T05:10:58.130 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:10:58.228 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T05:10:58.228 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T05:10:58.228 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:11:03.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-11T05:11:03.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-11T05:11:04.330 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T05:11:04.330 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T05:11:04.331 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:11:08.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-11T05:11:08.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-11T05:11:10.432 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T05:11:10.433 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T05:11:10.433 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T05:11:10.433 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:11:10.466 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T05:11:10.466 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T05:11:10.466 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T05:11:10.466 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:11:10.521 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T05:11:10.522 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T05:11:10.522 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:11:10.577 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T05:11:10.577 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T05:11:10.577 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:11:10.631 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T05:11:10.631 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T05:11:10.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-11T05:11:10.746 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T05:11:10.746 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T05:11:11.551 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T05:11:11.552 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T05:11:11.675 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T05:11:11.676 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T05:11:11.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T05:11:11.864 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T05:11:11.962 INFO:tasks.ceph:Archiving mon data... 2024-09-11T05:11:11.964 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176/data/mon.a.tgz 2024-09-11T05:11:11.964 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-11T05:11:11.987 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.0dUvKeGyvj 2024-09-11T05:11:11.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0dUvKeGyvj 2024-09-11T05:11:12.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.0dUvKeGyvj 2024-09-11T05:11:12.231 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.0dUvKeGyvj is 309KB 2024-09-11T05:11:12.328 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.0dUvKeGyvj 2024-09-11T05:11:12.344 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176/data/mon.b.tgz 2024-09-11T05:11:12.344 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-11T05:11:12.399 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.tKThNC8Ey9 2024-09-11T05:11:12.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.tKThNC8Ey9 2024-09-11T05:11:12.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.tKThNC8Ey9 2024-09-11T05:11:12.606 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.tKThNC8Ey9 is 314KB 2024-09-11T05:11:12.701 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.tKThNC8Ey9 2024-09-11T05:11:12.717 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176/data/mon.c.tgz 2024-09-11T05:11:12.717 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-11T05:11:12.772 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.J9ZzuHA4B3 2024-09-11T05:11:12.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.J9ZzuHA4B3 2024-09-11T05:11:12.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.J9ZzuHA4B3 2024-09-11T05:11:12.978 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.J9ZzuHA4B3 is 314KB 2024-09-11T05:11:13.073 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.J9ZzuHA4B3 2024-09-11T05:11:13.089 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T05:11:13.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T05:11:13.189 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T05:11:13.190 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T05:11:13.191 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176/remote/smithi031/crash 2024-09-11T05:11:13.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T05:11:13.258 INFO:tasks.ceph:Compressing logs... 2024-09-11T05:11:13.258 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T05:11:13.333 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T05:11:13.333 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46168.log 2024-09-11T05:11:13.333 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64116.log 2024-09-11T05:11:13.333 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.46168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52405.log 2024-09-11T05:11:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68526.log 2024-09-11T05:11:13.337 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64116.log.gz 2024-09-11T05:11:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39783.log 2024-09-11T05:11:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47620.log 2024-09-11T05:11:13.337 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68526.log.gz 2024-09-11T05:11:13.337 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44702.log 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59142.log 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64242.log/var/log/ceph/ceph-client.admin.44702.log: 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49969.log 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41463.log 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64242.log.gz 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61133.log 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49969.log: /var/log/ceph/ceph-client.admin.41463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68274.log 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61133.log.gz 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44362.log 2024-09-11T05:11:13.338 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68274.log.gz 2024-09-11T05:11:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44618.log 2024-09-11T05:11:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53077.log 2024-09-11T05:11:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58848.log 2024-09-11T05:11:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44618.log: /var/log/ceph/ceph-client.admin.53077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51061.log 2024-09-11T05:11:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49675.log 2024-09-11T05:11:13.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-11T05:11:13.583 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.51061.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55221.log 2024-09-11T05:11:13.583 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38092.log 2024-09-11T05:11:13.583 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59606.log 2024-09-11T05:11:13.583 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63275.log 2024-09-11T05:11:13.583 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38092.log.gz 2024-09-11T05:11:13.583 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.59606.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58302.log 2024-09-11T05:11:13.583 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5/var/log/ceph/ceph-client.admin.63275.log: --verbose -- /var/log/ceph/ceph-client.admin.70629.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63275.log.gz 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59522.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51187.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70629.log.gz 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64494.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46544.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56523.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64494.log.gz 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69872.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46544.log: gzip/var/log/ceph/ceph-client.admin.56523.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39244.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41352.log 2024-09-11T05:11:13.584 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69872.log.gz 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70881.log 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64326.log 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54506.log 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70881.log.gz 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38034.log 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64326.log.gz 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46695.log 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47620.log.gz 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53077.log.gz 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49969.log.gz 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52405.log.gz 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56523.log.gz 2024-09-11T05:11:13.585 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51061.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51187.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49675.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55221.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54506.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58260.log 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58848.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58302.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59606.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59522.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59142.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38034.log.gz 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60626.log 2024-09-11T05:11:13.586 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56817.log 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58260.log.gz 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.60626.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50137.log 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60626.log.gz 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56817.log.gz 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47361.log 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46991.log 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50137.log.gz 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73614.log 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47361.log.gz 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62099.log 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46991.log.gz 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.73614.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56355.log 2024-09-11T05:11:13.587 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73614.log.gz 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47176.log 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62099.log.gz 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67014.log 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56355.log.gz 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65880.log 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47176.log: /var/log/ceph/ceph-client.admin.67014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47509.log 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.47176.log.gz 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67014.log.gz 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38218.log 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65880.log.gz 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41894.log 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47509.log.gz 2024-09-11T05:11:13.588 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41278.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38218.log.gz 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57126.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46630.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58932.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57126.log.gz 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40005.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46630.log: /var/log/ceph/ceph-client.admin.58932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60755.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58932.log.gz 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62057.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52237.log 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60755.log.gz 2024-09-11T05:11:13.589 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62309.log 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62057.log: /var/log/ceph/ceph-client.admin.52237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62057.log.gz 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72856.log 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52237.log.gz 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66594.log 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62309.log.gz 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45118.log 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72856.log.gz 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66594.log.gz 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73276.log 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73571.log 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45118.log.gz 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2024-09-11T05:11:13.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73276.log.gz 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73571.log.gz 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.41241.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45664.log 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55389.log 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67854.log 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45664.log: /var/log/ceph/ceph-client.admin.55389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51313.log 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55389.log.gz 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39058.log 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67854.log.gz 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74707.log 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51313.log: /var/log/ceph/ceph-client.admin.39058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51313.log.gz 2024-09-11T05:11:13.591 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59100.log 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63149.log 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74707.log.gz 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41093.log 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59100.log.gz 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.63149.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51397.log 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63149.log.gz 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69493.log 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63486.log 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51397.log.gz 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70503.log 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69493.log.gz 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47398.log 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63486.log.gz 2024-09-11T05:11:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66006.log 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70503.log.gz 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63065.log 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47398.log.gz 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68863.log 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66006.log.gz 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46843.log 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63065.log.gz 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39894.log 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68863.log.gz 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65964.log 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46843.log.gz 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60405.log 2024-09-11T05:11:13.593 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64914.log 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65964.log.gz 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53035.log 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60405.log.gz 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60152.log 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64914.log.gz 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50851.log 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53035.log.gz 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51229.log 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60152.log.gz 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50683.log 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50851.log.gz 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39102.log 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51229.log.gz 2024-09-11T05:11:13.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50683.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39783.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66804.log 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39244.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50683.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40005.log.gz -- replaced with /var/log/ceph/ceph-client.admin.45664.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41463.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41352.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41093.log.gz 2024-09-11T05:11:13.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46544.log.gz 0.0% 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41278.log.gz 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51607.log 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46168.log.gz 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41894.log.gz 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46630.log.gz 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44702.log.gz -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44362.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44618.log.gz 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73740.log 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39102.log.gz 2024-09-11T05:11:13.596 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46695.log.gz 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66804.log.gz 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.51607.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57462.log 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51607.log.gz 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50389.log 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73740.log.gz 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68947.log 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57462.log.gz 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68442.log 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50389.log.gz 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63822.log 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68947.log.gz 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52615.log 2024-09-11T05:11:13.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68442.log.gz 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.63822.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64662.log 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63822.log.gz 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33626.log 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52615.log.gz 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43986.log 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64662.log.gz 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.33626.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38318.log 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33626.log.gz 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75085.log 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43986.log.gz 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51481.log 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38318.log.gz 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75085.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T05:11:13.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75085.log.gz 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44576.log 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51481.log.gz 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54842.log 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.44576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74749.log 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44576.log.gz 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73908.log 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54842.log.gz 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68569.log 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74749.log.gz 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39392.log 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73908.log.gz 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72394.log 2024-09-11T05:11:13.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68569.log.gz 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40301.log 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39392.log.gz 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72814.log 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72394.log.gz 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57756.log 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40301.log.gz 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67056.log 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr: 28.6% -- replaced with /var/log/ceph/ceph-client.admin.72814.log.gz 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74076.log/var/log/ceph/ceph-client.admin.57756.log: 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57756.log.gz 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55347.log 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67056.log.gz 2024-09-11T05:11:13.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60110.log 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74076.log.gz 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74959.log 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55347.log.gz 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66300.log 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60110.log.gz 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46806.log 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74959.log.gz 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60320.log 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66300.log.gz 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49213.log 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46806.log.gz 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72604.log 2024-09-11T05:11:13.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60320.log.gz 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49255.log 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49213.log.gz 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72015.log 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72604.log.gz 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55641.log 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49255.log.gz 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50515.log 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72015.log.gz 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47435.log 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55641.log.gz 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45874.log 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50515.log.gz 2024-09-11T05:11:13.602 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44596.log 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47435.log.gz 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45874.log.gz 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48625.log 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44596.log.gz 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48625.log.gz 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51985.log 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74160.log 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69577.log 2024-09-11T05:11:13.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51985.log.gz 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60026.log 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74160.log.gz 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56439.log 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69577.log.gz 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73192.log 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60026.log.gz 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40871.log 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56439.log.gz 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64032.log 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73192.log.gz 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40871.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40871.log.gz -- 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.53245.log 2024-09-11T05:11:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51691.log 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64032.log.gz 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58764.log 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53245.log.gz 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55851.log 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51691.log.gz 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53455.log 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58764.log.gz 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55515.log 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55851.log.gz 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48961.log 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53455.log.gz 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67644.log 2024-09-11T05:11:13.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55515.log.gz 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61847.log 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48961.log.gz 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67644.log.gz 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59016.log 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75167.log 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61847.log.gz 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55725.log 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.59016.log.gz 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59984.log 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75167.log.gz 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55725.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55725.log.gz 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47065.log 2024-09-11T05:11:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68905.log 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59984.log.gz 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59478.log 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47065.log.gz 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39328.log 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68905.log.gz 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49927.log 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59478.log.gz 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39328.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.39328.log.gz /var/log/ceph/ceph-client.admin.50221.log 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57924.log 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49927.log.gz 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46000.log 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50221.log.gz 2024-09-11T05:11:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52825.log 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57924.log.gz 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46000.log.gz 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50473.log 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69283.log 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52825.log.gz 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49129.log 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50473.log.gz 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59436.log 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69283.log.gz 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46769.log 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49129.log.gz 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73486.log 2024-09-11T05:11:13.608 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59436.log.gz 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69998.log 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46769.log.gz 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68611.log 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73486.log.gz 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65670.log 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69998.log.gz 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49087.log 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68611.log.gz 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50263.log 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65670.log.gz 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56958.log 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49087.log.gz 2024-09-11T05:11:13.609 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60068.log 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50263.log.gz 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56958.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.56958.log.gz --verbose 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.53666.log 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48751.log 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60068.log.gz 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70965.log 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53666.log.gz 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48751.log.gz 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38596.log 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73824.log 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70965.log.gz 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55263.log 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38596.log.gz 2024-09-11T05:11:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58596.log 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73824.log.gz 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72310.log 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55263.log.gz 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40811.log 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58596.log.gz 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63023.log 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72310.log.gz 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56145.log 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40811.log.gz 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50305.log 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63023.log.gz 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70335.log 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56145.log.gz 2024-09-11T05:11:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74497.log 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50305.log.gz 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46732.log 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70335.log.gz 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49633.log 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74497.log.gz 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55431.log 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46732.log.gz 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38658.log 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49633.log.gz 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45790.log 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55431.log.gz 2024-09-11T05:11:13.612 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40227.log 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38658.log.gz 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54926.log 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45790.log.gz 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37447.log 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40227.log.gz 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65124.log 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54926.log.gz 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60278.log 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37447.log.gz 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40908.log 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65124.log.gz 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58050.log 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60278.log.gz 2024-09-11T05:11:13.613 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64536.log 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40908.log.gz 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61301.log 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58050.log.gz 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71007.log 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64536.log.gz 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65292.log 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61301.log.gz 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46210.log 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71007.log.gz 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39820.log 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65292.log.gz 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.46210.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52783.log 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46210.log.gz 2024-09-11T05:11:13.614 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51859.log 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39820.log.gz 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41019.log 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52783.log.gz 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53287.log 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51859.log.gz 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38488.log 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41019.log.gz 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45076.log 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53287.log.gz 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51439.log 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38488.log.gz 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67518.log 2024-09-11T05:11:13.615 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45076.log.gz 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53119.log 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51439.log.gz 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47028.log 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67518.log.gz 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62141.log 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53119.log.gz 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63612.log 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47028.log.gz 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70587.log 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62141.log.gz 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60797.log 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63612.log.gz 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62645.log 2024-09-11T05:11:13.616 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70587.log.gz 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47287.log 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60797.log.gz 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46126.log 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62645.log.gz 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70461.log 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47287.log.gz 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62771.log 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46126.log.gz 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67938.log 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70461.log.gz 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56103.log 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62771.log.gz 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.67938.log: -- /var/log/ceph/ceph-client.admin.65586.log 2024-09-11T05:11:13.617 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67938.log.gz 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52531.log 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56103.log.gz 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55683.log 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65586.log.gz 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54800.log 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52531.log.gz 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55010.log 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55683.log.gz 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62183.log 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54800.log.gz 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68400.log 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55010.log.gz 2024-09-11T05:11:13.618 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38848.log 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62183.log.gz 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52951.log 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68400.log.gz 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58890.log 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38848.log.gz 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59184.log 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52951.log.gz 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58890.log.gz 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66426.log 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70125.log 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59184.log.gz 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70545.log 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66426.log.gz 2024-09-11T05:11:13.619 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33339.log 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70125.log.gz 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70545.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.70545.log.gz 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61511.log 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60965.log 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33339.log.gz 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64284.log 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61511.log.gz 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49297.log 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60965.log.gz 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52321.log 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64284.log.gz 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38657.log 2024-09-11T05:11:13.620 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49297.log.gz 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68232.log 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52321.log.gz 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44363.log 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38657.log.gz 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69746.log 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68232.log.gz 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60881.log 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44363.log.gz 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62519.log 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69746.log.gz 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44851.log 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60881.log.gz 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74034.log 2024-09-11T05:11:13.621 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62519.log.gz 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63696.log 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62981.log 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74034.log.gz 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59268.log 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63696.log.gz 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49759.log 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62981.log.gz 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74539.log 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59268.log.gz 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39968.log 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49759.log.gz 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67182.log 2024-09-11T05:11:13.622 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74539.log.gz 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64788.log 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39968.log.gz 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67182.log.gz 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60500.log 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49717.log 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64788.log.gz 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50809.log 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60500.log.gz 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73318.log 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49717.log.gz 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71259.log 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50809.log.gz 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53581.log 2024-09-11T05:11:13.623 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73318.log.gz 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44490.log 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71259.log.gz 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66636.log 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53581.log.gz 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49591.log 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44490.log.gz 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74917.log 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66636.log.gz 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68106.log 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49591.log.gz 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62393.log 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74917.log.gz 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53834.log 2024-09-11T05:11:13.624 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68106.log.gz 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68989.log 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62393.log.gz 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45454.log 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53834.log.gz 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42840.log 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68989.log.gz 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57630.log 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45454.log.gz 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56229.log 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42840.log.gz 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50725.log 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57630.log.gz 2024-09-11T05:11:13.625 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69704.log 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56229.log.gz 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72478.log 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50725.log.gz 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55599.log 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69704.log.gz 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40769.log 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72478.log.gz 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69115.log 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55599.log.gz 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56775.log 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40769.log.gz 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40834.log 2024-09-11T05:11:13.626 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69115.log.gz 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51775.log 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56775.log.gz 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65796.log 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40834.log.gz 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71427.log 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51775.log.gz 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61805.log 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65796.log.gz 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54380.log 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71427.log.gz 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53792.log 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61805.log.gz 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54380.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T05:11:13.627 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54380.log.gz 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38764.log 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53792.log.gz 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52027.log 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70209.log 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38764.log.gz 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65082.log 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52027.log.gz 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5/var/log/ceph/ceph-client.admin.70209.log: --verbose -- /var/log/ceph/ceph-client.admin.73402.log 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70209.log.gz 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66762.log 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65082.log.gz 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43859.log 2024-09-11T05:11:13.628 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73402.log.gz 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71133.log 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66762.log.gz 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73950.log 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43859.log.gz 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66510.log 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71133.log.gz 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70377.log 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73950.log.gz 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5/var/log/ceph/ceph-client.admin.66510.log: --verbose -- /var/log/ceph/ceph-client.admin.53203.log 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66510.log.gz 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70377.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.71763.log 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70377.log.gz 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53329.log 2024-09-11T05:11:13.629 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53203.log.gz 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59352.log 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71763.log.gz 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57672.log 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53329.log.gz 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59352.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59352.log.gz 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41759.log 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66090.log 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57672.log.gz 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41759.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41759.log.gz --verbose 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.52363.log 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71049.log 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66090.log.gz 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59942.log 2024-09-11T05:11:13.630 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52363.log.gz 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33419.log 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71049.log.gz 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69620.log 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59942.log.gz 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50641.log 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33419.log.gz 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67812.log 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69620.log.gz 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40727.log 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50641.log.gz 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57000.log 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67812.log.gz 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70671.log 2024-09-11T05:11:13.631 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40727.log.gz 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72268.log 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57000.log.gz 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62855.log 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70671.log.gz 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55095.log 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72268.log.gz 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39857.log 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62855.log.gz 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64620.log 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55095.log.gz 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51355.log 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39857.log.gz 2024-09-11T05:11:13.632 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61007.log 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64620.log.gz 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39243.log 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51355.log.gz 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68064.log 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61007.log.gz 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51649.log 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39243.log.gz 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50179.log 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68064.log.gz 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51649.log.gz 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55977.log 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50179.log.gz 2024-09-11T05:11:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41685.log 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58974.log 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55977.log.gz 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44744.log 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41685.log.gz 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64074.log 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58974.log.gz 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61889.log 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44744.log.gz 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60584.log 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64074.log.gz 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70041.log 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61889.log.gz 2024-09-11T05:11:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41611.log 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60584.log.gz 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72940.log/var/log/ceph/ceph-client.admin.70041.log: 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70041.log.gz 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47472.log 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41611.log.gz 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72142.log 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72940.log.gz 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52573.log 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47472.log.gz 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38616.log 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72142.log.gz 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67728.log 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52573.log.gz 2024-09-11T05:11:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52195.log 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38616.log.gz 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56903.log 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67728.log.gz 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65922.log 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52195.log.gz 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54674.log 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56903.log.gz 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64410.log 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65922.log.gz 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69367.log 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54674.log.gz 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63107.log 2024-09-11T05:11:13.636 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64410.log.gz 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43860.log 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69367.log.gz 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38176.log 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63107.log.gz 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45706.log 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43860.log.gz 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55137.log 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38176.log.gz 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71343.log 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45706.log.gz 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57294.log 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55137.log.gz 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39168.log 2024-09-11T05:11:13.637 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71343.log.gz 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54968.log 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57294.log.gz 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60839.log 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39168.log.gz 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71091.log 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54968.log.gz 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44828.log 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60839.log.gz 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71847.log 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71091.log.gz 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39644.log 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44828.log.gz 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49045.log 2024-09-11T05:11:13.638 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71847.log.gz 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54590.log 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39644.log.gz 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68737.log 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49045.log.gz 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65712.log 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54590.log.gz 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33537.log 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68737.log.gz 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72436.log 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65712.log.gz 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67770.log 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33537.log.gz 2024-09-11T05:11:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67266.log 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72436.log.gz 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33299.log 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67770.log.gz 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72057.log 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67266.log.gz 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73656.log 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54548.log 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72057.log.gz 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62225.log 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73656.log.gz 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64830.log 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54548.log.gz 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53161.log 2024-09-11T05:11:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62225.log.gz 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62435.log 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64830.log.gz 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48919.log 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53161.log.gz 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60671.log 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62435.log.gz 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39746.log 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48919.log.gz 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.60671.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.61721.log 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60671.log.gz 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.33299.log.gz 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71553.log 2024-09-11T05:11:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39746.log.gz 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68148.log 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61721.log.gz 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73992.log 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71553.log.gz 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63948.log 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68148.log.gz 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73108.log 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73992.log.gz 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64746.log 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63948.log.gz 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59564.log 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73108.log.gz 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75001.log 2024-09-11T05:11:13.642 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64746.log.gz 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56649.log 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59564.log.gz 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65334.log 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75001.log.gz 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70083.log 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56649.log.gz 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68484.log 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65334.log.gz 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70083.log.gz 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72184.log 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49381.log 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68484.log.gz 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70167.log 2024-09-11T05:11:13.643 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72184.log.gz 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54044.log 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49381.log.gz 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62015.log 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70167.log.gz 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67140.log 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54044.log.gz 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40116.log 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62015.log.gz 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74665.log 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67140.log.gz 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66678.log 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40116.log.gz 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64200.log 2024-09-11T05:11:13.644 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74665.log.gz 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44450.log 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66678.log.gz 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58428.log 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64200.log.gz 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74202.log 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44450.log.gz 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56859.log 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58428.log.gz 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55935.log 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74202.log.gz 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72688.log 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56859.log.gz 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73066.log 2024-09-11T05:11:13.645 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55935.log.gz 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61175.log 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72688.log.gz 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73066.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73066.log.gz 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53708.log 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61175.log.gz 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53918.log 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.53708.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.69073.log 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53708.log.gz 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61217.log 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53918.log.gz 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73444.log 2024-09-11T05:11:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69073.log.gz 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44722.log 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61217.log.gz 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45244.log 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73444.log.gz 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40412.log 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44722.log.gz 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45412.log 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45244.log.gz 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59310.log 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40412.log.gz 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72646.log 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45412.log.gz 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69241.log 2024-09-11T05:11:13.647 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59310.log.gz 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48667.log 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72646.log.gz 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41796.log 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69241.log.gz 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69157.log 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48667.log.gz 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41796.log.gz 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61931.log 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47102.log 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69157.log.gz 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66468.log 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61931.log.gz 2024-09-11T05:11:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74581.log 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47102.log.gz 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66468.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.66468.log.gz 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.71469.log 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71595.log 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74581.log.gz 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55557.log 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71469.log.gz 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57714.log 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71595.log.gz 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72982.log 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55557.log.gz 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71511.log 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57714.log.gz 2024-09-11T05:11:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.72982.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52699.log 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72982.log.gz 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66384.log 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71511.log.gz 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52699.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.52699.log.gz --verbose 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.45580.log 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56481.log 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66384.log.gz 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44240.log 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45580.log.gz 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55052.log 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56481.log.gz 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38448.log 2024-09-11T05:11:13.650 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44240.log.gz 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65208.log 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55052.log.gz 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46286.log 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38448.log.gz 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61679.log 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65208.log.gz 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71931.log 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46286.log.gz 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70797.log 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61679.log.gz 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52741.log 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71931.log.gz 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43946.log 2024-09-11T05:11:13.651 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70797.log.gz 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57966.log 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52741.log.gz 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr: 94.9% -- replaced with /var/log/ceph/ceph-client.admin.44851.log.gz 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52657.log 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43946.log.gz 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39476.log 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57966.log.gz 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70923.log 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52657.log.gz 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50431.log 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39476.log.gz 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33162.log 2024-09-11T05:11:13.652 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70923.log.gz 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55179.log 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50431.log.gz 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49801.log 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33162.log.gz 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60713.log 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55179.log.gz 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47657.log 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49801.log.gz 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65544.log 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60713.log.gz 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47657.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47657.log.gz -5 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52909.log 2024-09-11T05:11:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58092.log 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65544.log.gz 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59816.log 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52909.log.gz 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58092.log.gz 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54464.log 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63570.log 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59816.log.gz 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39560.log 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54464.log.gz 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70839.log 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63570.log.gz 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68653.log 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39560.log.gz 2024-09-11T05:11:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74875.log 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70839.log.gz 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58344.log 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68653.log.gz 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74875.log.gz 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38806.log 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69830.log 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58344.log.gz 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66048.log 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38806.log.gz 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41167.log 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69830.log.gz 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66048.log.gz 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39686.log 2024-09-11T05:11:13.655 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41167.log.gzgzip 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72520.log 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40153.log 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39686.log.gz 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71637.log 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72520.log.gz 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61427.log 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40153.log.gz 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57168.log 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71637.log.gz 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43985.log 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61427.log.gz 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71217.log 2024-09-11T05:11:13.656 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57168.log.gz 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44092.log 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43985.log.gz 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54884.log 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71217.log.gz 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49003.log 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44092.log.gz 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54884.log.gz 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61973.log 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57504.log 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49003.log.gz 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49171.log 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61973.log.gz 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58554.log 2024-09-11T05:11:13.657 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57504.log.gz 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58680.log 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49171.log.gz 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58554.log.gz 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68316.log 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74623.log/var/log/ceph/ceph-client.admin.58680.log: 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58680.log.gz 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39602.log 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68316.log.gz 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74623.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74623.log.gz 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49843.log 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39602.log.gz 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65376.log 2024-09-11T05:11:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44282.log 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49843.log.gz 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65376.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.65376.log.gz /var/log/ceph/ceph-client.admin.73782.log 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61469.log 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44282.log.gz 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54212.log 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73782.log.gz 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45328.log 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61469.log.gz 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44072.log 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54212.log.gz 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64452.log 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45328.log.gz 2024-09-11T05:11:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59058.log 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44072.log.gz 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64452.log.gz 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46917.log 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59058.log.gz 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63906.log 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65838.log 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46917.log.gz 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54758.log 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63906.log.gz 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54170.log 2024-09-11T05:11:13.660 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65838.log.gz 2024-09-11T05:11:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74371.log 2024-09-11T05:11:13.661 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54758.log.gz 2024-09-11T05:11:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51565.log 2024-09-11T05:11:13.661 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54170.log.gz 2024-09-11T05:11:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52153.log 2024-09-11T05:11:13.661 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74371.log.gz 2024-09-11T05:11:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69409.log 2024-09-11T05:11:13.664 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51565.log.gz 2024-09-11T05:11:13.664 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68695.log 2024-09-11T05:11:13.664 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52153.log.gz 2024-09-11T05:11:13.664 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45496.log 2024-09-11T05:11:13.664 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69409.log.gz 2024-09-11T05:11:13.664 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50767.log 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68695.log.gz 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60447.log 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45496.log.gz 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69788.log 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50767.log.gz 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54422.log 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60447.log.gz 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54296.log 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69788.log.gz 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71385.log 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54422.log.gz 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47324.log 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54296.log.gz 2024-09-11T05:11:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59226.log 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71385.log.gz 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47324.log.gz 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70419.log 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59226.log.gz 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65754.log 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73698.log 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70419.log.gz 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58512.log 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65754.log.gz 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55473.log 2024-09-11T05:11:13.666 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73698.log.gz 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58512.log.gz 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.55473.log: -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55473.log.gz 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61385.log 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53624.log 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59690.log 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61385.log.gz 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62561.log 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53624.log.gz 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46880.log 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59690.log.gz 2024-09-11T05:11:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57420.log 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62561.log.gz 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.46880.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.72100.log 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46880.log.gz 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57420.log.gz 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45034.log 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62267.log 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72100.log.gz 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73866.log 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66342.log 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62267.log.gz 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62351.log 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73866.log.gz 2024-09-11T05:11:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64704.log 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66342.log.gz 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57210.log 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62351.log.gz 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54086.log 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64704.log.gz 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73150.log 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57210.log.gz 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58386.log 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.45034.log.gz 0.0% 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54086.log.gz 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67980.log 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73150.log.gz 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69956.log 2024-09-11T05:11:13.669 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58386.log.gz 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69662.log 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67980.log.gz 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67686.log 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69956.log.gz 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71973.log 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69662.log.gz 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51271.log 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67686.log.gz 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61637.log 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71973.log.gz 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55305.log 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51271.log.gz 2024-09-11T05:11:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49465.log 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61637.log.gz 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71721.log 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55305.log.gz 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43858.log 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49465.log.gz 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38319.log 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71721.log.gz 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43858.log.gz 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57252.log 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51817.log 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38319.log.gz 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45832.log 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57252.log.gz 2024-09-11T05:11:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50893.log 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51817.log.gz 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45916.log 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45832.log.gz 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50893.log: -- /var/log/ceph/ceph-client.admin.73360.log 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50893.log.gz 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58218.log 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45916.log.gz 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65166.log 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73360.log.gz 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67308.log 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58218.log.gz 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56397.log 2024-09-11T05:11:13.672 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65166.log.gz 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50557.log 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67308.log.gz 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.56397.log: /var/log/ceph/ceph-client.admin.54716.log 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56397.log.gz 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56733.log 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50557.log.gz 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65460.log 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54716.log.gz 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51523.log 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56733.log.gz 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65460.log.gz 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40982.log 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38890.log 2024-09-11T05:11:13.673 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51523.log.gz 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40982.log.gz 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56313.log 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52489.log 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38890.log.gz 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56313.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.56313.log.gz --verbose 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph.tmp-client.admin.27259.log 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71805.log 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52489.log.gz 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.27259.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27259.log.gz 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45958.log 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68022.log/var/log/ceph/ceph-client.admin.71805.log: 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71805.log.gz 2024-09-11T05:11:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56271.log 0.0% 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45958.log.gz 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70293.log 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68022.log.gz 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62813.log 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56271.log.gz 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44992.log 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70293.log.gz 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40042.log 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62813.log.gz 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61343.log 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44992.log.gz 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50935.log 2024-09-11T05:11:13.675 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40042.log.gz 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45538.log 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61343.log.gz 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75127.log 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50935.log.gz 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44364.log 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45538.log.gz 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57798.log 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75127.log.gz 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44364.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44364.log.gz 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61553.log 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57798.log.gz 2024-09-11T05:11:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45370.log 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61553.log.gz 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53413.log 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74413.log 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45370.log.gz 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63780.log 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53413.log.gz 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40449.log 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74413.log.gz 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59732.log 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63780.log.gz 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68821.log 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40449.log.gz 2024-09-11T05:11:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66132.log 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59732.log.gz 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74244.log 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68821.log.gz 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69031.log 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66132.log.gz 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49507.log 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74244.log.gz 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69031.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.61049.log -- replaced with /var/log/ceph/ceph-client.admin.69031.log.gz 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70755.log 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49507.log.gz 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61049.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.61049.log.gz /var/log/ceph/ceph-client.admin.45160.log 2024-09-11T05:11:13.678 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41204.log 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70755.log.gz 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47583.log 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45160.log.gz 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40338.log 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41204.log.gz 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65250.log 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47583.log.gz 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73024.log 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40338.log.gz 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54338.log 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65250.log.gz 2024-09-11T05:11:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64872.log 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73024.log.gz 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66846.log 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54338.log.gz 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58470.log 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr: 95.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64872.log.gz 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74791.log 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66846.log.gz 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59774.log 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58470.log.gz 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46507.log 2024-09-11T05:11:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74791.log.gz 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61259.log 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59774.log.gz 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46954.log 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46507.log.gz 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71175.log 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61259.log.gz 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43778.log 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46954.log.gz 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67098.log 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71175.log.gz 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48709.log 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43778.log.gz 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54002.log 2024-09-11T05:11:13.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67098.log.gz 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48709.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48709.log.gz 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52867.log 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54002.log.gz 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44132.log 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52867.log.gz 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62687.log 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41130.log 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44132.log.gz 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62687.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.46418.log -- replaced with /var/log/ceph/ceph-client.admin.62687.log.gz 2024-09-11T05:11:13.682 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57378.log 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41130.log.gz 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46418.log.gz 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40671.log 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44950.log 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57378.log.gz 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54128.log 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40671.log.gz 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71889.log 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44950.log.gz 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38317.log 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54128.log.gz 2024-09-11T05:11:13.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51943.log 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71889.log.gz 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45622.log 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38317.log.gz 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38447.log 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51943.log.gz 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67434.log 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45622.log.gz 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47250.log 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38447.log.gz 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66216.log 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67434.log.gz 2024-09-11T05:11:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67602.log 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47250.log.gz 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39931.log 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66216.log.gz 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71301.log 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67602.log.gz 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69325.log 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39931.log.gz 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50599.log 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71301.log.gz 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69325.log.gz 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40264.log 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44908.log 2024-09-11T05:11:13.685 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50599.log.gz 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66258.log 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40264.log.gz 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62603.log 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44908.log.gz 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51901.log 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66258.log.gz 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38932.log 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62603.log.gz 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41722.log 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51901.log.gz 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46042.log 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38932.log.gz 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45748.log 2024-09-11T05:11:13.686 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41722.log.gz 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67392.log 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46042.log.gz 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63317.log 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45748.log.gz 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74287.log 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67392.log.gz 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48835.log 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63317.log.gz 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53539.log 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74287.log.gz 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68358.log 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48835.log.gz 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57882.log 2024-09-11T05:11:13.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53539.log.gz 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63191.log 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68358.log.gz 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63233.log 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57882.log.gz 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51019.log 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63191.log.gz 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74329.log 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63233.log.gz 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60923.log 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51019.log.gz 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74455.log 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74329.log.gz 2024-09-11T05:11:13.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60923.log.gz 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41648.log 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32986.log 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74455.log.gz 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62939.log 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41648.log.gz 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.32986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45286.log 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32986.log.gz 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61091.log 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62939.log.gz 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64158.log 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45286.log.gz 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71679.log 2024-09-11T05:11:13.689 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61091.log.gz 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69914.log 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64158.log.gz 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.71679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59394.log 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71679.log.gz 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73234.log 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69914.log.gz 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41426.log 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59394.log.gz 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73234.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73234.log.gz 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46527.log 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41426.log.gz 2024-09-11T05:11:13.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39412.log 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46527.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46527.log.gz 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63654.log 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38974.log 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39412.log.gz 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66888.log 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63654.log.gz 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57840.log 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38974.log.gz 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63402.log 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66888.log.gz 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56565.log 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57840.log.gz 2024-09-11T05:11:13.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65418.log 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63402.log.gz 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58008.log 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56565.log.gz 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58638.log 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65418.log.gz 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58134.log 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58008.log.gz 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59648.log 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58638.log.gz 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52447.log 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58134.log.gz 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58806.log 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59648.log.gz 2024-09-11T05:11:13.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75043.log 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52447.log.gz 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41500.log 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58806.log.gz 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.75043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75043.log.gz 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50011.log 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41500.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.52279.log 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41500.log.gz 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64998.log 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50011.log.gz 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67476.log 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52279.log.gz 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63738.log 2024-09-11T05:11:13.693 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64998.log.gz 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58176.log 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67476.log.gz 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70713.log 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63738.log.gz 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58176.log.gz 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64956.log 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.70713.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.61763.log 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70713.log.gz 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55893.log 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64956.log.gz 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51145.log 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61763.log.gz 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55893.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.55893.log.gz --verbose 2024-09-11T05:11:13.694 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.39016.log 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48793.log 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51145.log.gz 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67224.log 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41315.log 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48793.log.gz 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46291.log 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67224.log.gz 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72730.log 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41315.log.gz 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64578.log 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46291.log.gz 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66174.log 2024-09-11T05:11:13.695 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72730.log.gz 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49423.log 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64578.log.gz 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38134.log 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66174.log.gz 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74833.log 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49423.log.gz 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70251.log 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38134.log.gz 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50977.log 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74833.log.gz 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.70251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60362.log 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70251.log.gz 2024-09-11T05:11:13.696 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54254.log 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50977.log.gz 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60236.log 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60362.log.gz 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55809.log 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54254.log.gz 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58722.log 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60236.log.gz 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63528.log 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55809.log.gz 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67560.log 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58722.log.gz 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46672.log 2024-09-11T05:11:13.697 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63528.log.gz 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68779.log 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67560.log.gz 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62477.log 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46672.log.gz 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47546.log 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68779.log.gz 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.62477.log: /var/log/ceph/ceph-client.admin.56019.log 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62477.log.gz 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51733.log 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47546.log.gz 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56019.log.gz 2024-09-11T05:11:13.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51733.log.gz 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63359.log 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65040.log 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46084.log 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63359.log.gz 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65040.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.65040.log.gz 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.44489.log 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33379.log 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46084.log.gz 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62897.log 2024-09-11T05:11:13.699 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44489.log.gz 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57588.log 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33379.log.gz 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72898.log 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62897.log.gz 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53497.log 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57588.log.gz 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51103.log 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72898.log.gz 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68190.log 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53497.log.gz 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50095.log 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51103.log.gz 2024-09-11T05:11:13.700 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.68190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57042.log 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68190.log.gz 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44198.log 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50095.log.gz 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65502.log 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57042.log.gz 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72352.log 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44198.log.gz 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52069.log 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65502.log.gz 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45202.log 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72352.log.gz 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52069.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.52069.log.gz /var/log/ceph/ceph-client.admin.62729.log 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66930.log 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45202.log.gz 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62729.log.gz 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47139.log 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55767.log 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66930.log.gz 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59858.log 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47139.log.gz 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67350.log 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55767.log.gz 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65628.log 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59858.log.gz 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49339.log 2024-09-11T05:11:13.702 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67350.log.gz 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53371.log 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65628.log.gz 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57336.log 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49339.log.gz 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40523.log 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53371.log.gz 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66552.log 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57336.log.gz 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40945.log 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40523.log.gz 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69451.log 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66552.log.gz 2024-09-11T05:11:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38487.log 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40945.log.gz 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69535.log 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69451.log.gz 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53960.log 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38487.log.gz 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57084.log 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69535.log.gz 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59900.log 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53960.log.gz 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63990.log 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57084.log.gz 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59900.log.gz 2024-09-11T05:11:13.704 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72772.log 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63990.log.gz 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41389.log 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46378.log 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72772.log.gz 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53750.log 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41389.log.gz 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46378.log.gz 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49549.log 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53750.log.gz 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50053.log 2024-09-11T05:11:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52993.log 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49549.log.gz 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63444.log 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50053.log.gz 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48877.log 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52993.log.gz 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38320.log 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63444.log.gz 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66972.log 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48877.log.gz 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53876.log 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38320.log.gz 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66972.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.40190.log -- replaced with /var/log/ceph/ceph-client.admin.66972.log.gz 2024-09-11T05:11:13.706 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56187.log 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53876.log.gz 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72562.log 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40190.log.gz 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74118.log 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56187.log.gz 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72562.log.gz 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46416.log 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56691.log 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.74118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74118.log.gz 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61595.log 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46416.log.gz 2024-09-11T05:11:13.707 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49885.log 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56691.log.gz 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72226.log 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61595.log.gz 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64368.log 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49885.log.gz 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.72226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54632.log 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72226.log.gz 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66720.log 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64368.log.gz 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63864.log 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54632.log.gz 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56607.log 2024-09-11T05:11:13.708 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66720.log.gz 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60194.log 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63864.log.gz 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56607.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.56607.log.gz -- 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.73528.log 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47213.log 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60194.log.gz 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.73528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69199.log 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73528.log.gz 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57546.log 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47213.log.gz 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.69199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56061.log 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69199.log.gz 2024-09-11T05:11:13.709 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57546.log.gz 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67896.log 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56061.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.56061.log.gz -- 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.50347.log 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60542.log 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67896.log.gz 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46292.log 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50347.log.gz 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52111.log 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60542.log.gz 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46292.log.gz 2024-09-11T05:11:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52111.log.gz 2024-09-11T05:11:13.711 INFO:teuthology.orchestra.run.smithi031.stderr: 96.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T05:11:14.371 INFO:teuthology.orchestra.run.smithi031.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T05:11:16.894 INFO:teuthology.orchestra.run.smithi031.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-11T05:11:18.234 INFO:teuthology.orchestra.run.smithi031.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-11T05:11:20.178 INFO:teuthology.orchestra.run.smithi031.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T05:25:17.015 INFO:teuthology.orchestra.run.smithi031.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T05:26:57.300 INFO:teuthology.orchestra.run.smithi031.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T05:28:17.460 INFO:teuthology.orchestra.run.smithi031.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T05:28:19.415 INFO:teuthology.orchestra.run.smithi031.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T05:28:19.417 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T05:28:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:real 17m6.104s 2024-09-11T05:28:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:user 6m49.451s 2024-09-11T05:28:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m18.327s 2024-09-11T05:28:19.417 INFO:tasks.ceph:Archiving logs... 2024-09-11T05:28:19.418 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176/remote/smithi031/log 2024-09-11T05:28:19.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T05:29:26.650 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T05:29:26.661 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T05:29:26.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T05:29:26.772 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T05:29:26.780 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T05:29:26.780 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:29:26.803 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-09-11T05:29:26.848 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T05:29:26.848 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-09-11T05:29:26.848 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com 3 6 377 47 +197us[ +194us] +/- 43ms 2024-09-11T05:29:26.849 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -1125us[-1125us] +/- 75ms 2024-09-11T05:29:26.849 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +942us[ +939us] +/- 87ms 2024-09-11T05:29:26.849 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T05:29:26.849 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T05:29:26.858 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T05:29:26.858 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T05:29:26.866 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T05:29:27.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T05:29:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:29:27.078 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T05:29:27.086 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T05:29:27.129 INFO:teuthology.task.internal:Duration was 2481.599649 seconds 2024-09-11T05:29:27.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T05:29:27.191 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T05:29:27.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T05:29:27.254 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T05:29:27.540 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T05:29:27.540 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T05:29:27.540 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T05:29:27.564 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T05:29:27.565 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T05:29:27.634 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T05:29:27.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T05:29:27.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T05:29:27.831 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T05:29:27.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T05:29:27.898 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T05:29:27.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:29:27.940 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-09-11T05:29:27.957 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:29:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:29:28.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T05:29:28.023 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T05:29:28.024 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900176/remote/smithi031 2024-09-11T05:29:28.024 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T05:29:28.092 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T05:29:28.092 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T05:29:28.143 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T05:29:28.153 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T05:29:28.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T05:29:28.162 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T05:29:28.163 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T05:29:28.198 INFO:teuthology.orchestra.run.smithi031.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 05:29 /home/ubuntu/cephtest 2024-09-11T05:29:28.199 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T05:29:28.211 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T05:29:28.234 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest}} duration: 2481.5996487140656 flavor: default owner: scheduled_hyelloji@teuthology success: true 2024-09-11T05:29:28.235 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T05:29:28.312 INFO:teuthology.run:pass